acceptodds
Withdrawn from ICLR 2027

CryoTracer: A Synthetic-to-Real System for Filament Instance Tracing in Cryo-EM

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Particle picking in cryo-electron microscopy is commonly formulated as an object-detection problem in which individual particles are represented by bounding boxes, a natural representation for globular particles but a poor match for filaments that can extend across an entire micrograph. Existing methods reconstruct filament trajectories by linking local predictions in a separate post-processing stage, decoupling detection from instance formation and creating potential ambiguities at gaps and crossings. Directly preserving filament identity is particularly useful for polymorphic specimens, where segments originating from the same filament should remain associated. However, learning such instance-level representations is challenging because expert centerline annotations are scarce and costly to obtain. We introduce CryoTracer, a synthetic-to-real framework that addresses this limitation through procedural micrograph generation, transformer-based polyline prediction, and specimen-specific fine-tuning. Procedural pretraining exposes the model to diverse filament geometries, intersections, and imaging conditions before adaptation using a limited number of fully annotated experimental micrographs. CryoTracer represents filaments as ordered centerline polylines and optimizes their geometry using an orientation-invariant discrete Fréchet objective, enabling consistent learning regardless of tracing direction. Evaluation on established and newly expert-annotated datasets shows that CryoTracer improves instance-level filament recovery over existing particle-detection approaches. Complementary metrics distinguish regional coverage from complete-instance recovery, providing a more detailed assessment of filament-tracing performance. Together, these results establish end-to-end polyline prediction as a practical alternative to detection-and-linking pipelines for recovering complete filament instances with limited micrograph annotation budgets.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
computer vision, object detection, cryo-em

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jbU8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents CryoTracer, a method to trace filaments in cryo-EM images. Each filament is predicted directly as an ordered line with five points, using an adapted RF-DETR. The loss is based on the Fréchet distance and does not depend on the direction of the line. The model is first trained on synthetic images and then fine-tuned on real annotations. On two datasets (TMV and an in-house α-synuclein dataset) it clearly outperforms Topaz, crYOLO and the RELION AmyTracer. The authors also release new annotations for EMPIAR-10022.

Critical strengths
  1. The paper brings direct polyline prediction, known from map and lane detection, to filament picking in cryo-EM. Together with the synthetic pretraining it clearly improves over the particle pickers that are used in cryo-EM pipelines today. The idea itself is not new, but the transfer to this domain works well.
  2. The ablations are well done and show the effect of the synthetic pretraining, the Fréchet loss and the architecture choices. The new annotations for EMPIAR-10022 are useful for the community.

Further strengths:

  • The setting with only a few annotated images is realistic.
  • Code is provided and the main results are averaged over several seeds.
Critical weaknesses
  1. Important filament tracing methods are missing. Most important is TARDIS [1], which does instance segmentation of filaments in electron microscopy images and is open source. Its pretrained models are developed and demonstrated mainly on cryo-ET tomograms (and TIRF fluorescence), so it is not clear whether they transfer to 2D cryo-EM micrographs such as those used here. It should still be included as a baseline, as was done for AmyTracer, or the paper should explain why it cannot be used. Other established tracing methods are also neither discussed nor compared: SOAX [2], SIFNE [3], FIESTA [4], MTrack [5], Liu et al. [6–8], DRIFT [9], Masoudi et al. [10] and KnotResolver [11]. The current baselines are mostly particle pickers, and the metric rewards complete filaments, so it is unclear how large the gain over real tracing methods is.
  2. It is not shown that the method generalizes. Only two datasets are used, and one of them is private. On α-synuclein, images from the same recording session are in the training and in the test set. They share the sample and the recording conditions.

Further weaknesses (can be fixed in a revision):

  • The related work also misses other relevant areas.
    • Synthetic training data for microscopy, for example AnyStar [12], CARE [13], MicSim_FluoMT [14], biobeam [15] and SynthMT [16]. This matters here, because the model trained only on synthetic data does not work at all on real data. So the gap between the synthetic and the real images is large.
    • Generalist segmentation models like SAM 3 [19], and their microscopy variants such as µSAM [20], CellSAM [21] and Cellpose-SAM [22]. SAM 3 with a text prompt would be an easy additional baseline.
    • Polyline prediction outside microscopy.

Without this context, the novelty of the method is overstated.

[1] R. Kiewisz et al. Accurate and fast segmentation of filaments and membranes in micrographs and tomograms with TARDIS. bioRxiv, 2024.

[2] T. Xu et al. SOAX: a software for quantification of 3D biopolymer networks. Scientific Reports, 2015.

[3] Z. Zhang, Y. Nishimura, P. Kanchanawong. Extracting microtubule networks from superresolution single-molecule localization microscopy data. Molecular Biology of the Cell, 2017.

[4] F. Ruhnow, D. Zwicker, S. Diez. Tracking single particles and elongated filaments with nanometer precision. Biophysical Journal, 2011.

[5] V. Kapoor et al. MTrack: automated detection, tracking, and analysis of dynamic microtubules. Scientific Reports, 2019.

[6] Y. Liu et al. Densely connected stacked U-network for filament segmentation in microscopy images. ECCV Workshops, 2018.

[7] Y. Liu et al. Intersection to overpass: instance segmentation on filamentous structures with an orientation-aware neural network and terminus pairing algorithm. CVPR Workshops, 2019.

[8] Y. Liu, J. Caplan, C. Kambhamettu. Extraction and quantification of actin cytoskeleton in microscopic images using a deep learning based framework and a curve clustering model. ICPR, 2022.

[9] Y. Liu, S. Peng, J. Caplan, C. Kambhamettu. Pick and trace: instance segmentation for filamentous objects with a recurrent neural network. MICCAI, 2023.

[10] S. Masoudi et al. Instance-level microtubule tracking. IEEE Transactions on Medical Imaging, 2020.

[11] D. Khatri, S. A. Yadav, C. A. Athale. KnotResolver: tracking self-intersecting filaments in microscopy using directed graphs. Bioinformatics, 2024.

[12] N. Dey et al. AnyStar: domain randomized universal star-convex 3D instance segmentation. WACV, 2024.

[13] M. Weigert et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nature Methods, 2018.

[14] A. A. Laydi et al. Adaptive attention residual U-Net for curvilinear structure segmentation in fluorescence microscopy and biomedical images. arXiv, 2025.

[15] M. Weigert et al. Biobeam: multiplexed wave-optical simulations of light-sheet microscopy. PLOS Computational Biology, 2018.

[16] M. Koddenbrock et al. Synthetic data enables human-grade microtubule analysis with foundation models for segmentation. PLOS Computational Biology, 2026.

[19] N. Carion et al. SAM 3: segment anything with concepts. arXiv, 2025.

[20] A. Archit et al. Segment anything for microscopy. Nature Methods, 2025.

[21] M. Marks et al. CellSAM: a foundation model for cell segmentation. Nature Methods, 2025.

[22] M. Pachitariu, M. Rariden, C. Stringer. Cellpose-SAM: superhuman generalization for cellular segmentation. bioRxiv, 2025.

Questions for authors

Clarification and additional analysis (the answers could change my assessment):

  • How does CryoTracer compare to TARDIS [1] on your datasets? If TARDIS cannot be used on 2D cryo-EM images, please explain why.
  • How much of the improvement on α-synuclein comes from images of the same session in training and test? Can you report results with a split by session?
  • How exactly are the Topaz outputs converted into filaments? As I understand it, each filament becomes a straight segment between its end points, while the ground truth can be curved. What is the effect of this endpoint interpolation on the Topaz scores? Would the ranking change if the traces followed the picked coordinates?

Suggestions (not part of my assessment):

  • Please report Skeleton IoU (SKIoU [6,16]), or explain why it does not fit. It compares the center lines directly and does not depend on the chosen box width like Box IoU does. It can be computed on your existing predictions without retraining.
  • The baselines were run once, while CryoTracer is averaged over five seeds. Reporting the variation for the baselines as well would make the comparison fairer.
  • Five points per filament is not enough for strongly curved filaments, as you note yourselves. A flexible number of points or a spline could help.
Reviewer_36KqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents CryoTracer, a synthetic-to-real framework for tracing individual filaments in cryo-EM micrographs. Particle-picking pipelines usually detect local segments and then link them into trajectories in post-processing. CryoTracer instead predicts each filament directly as an ordered centerline polyline. The architecture builds on RF-DETR, adding polyline proposals that are refined iteratively with deformable attention, and the geometric loss is an orientation-invariant discrete Fréchet distance. To reduce the need for manual annotation, the model is first pretrained on procedurally generated micrographs and then fine-tuned on annotations from the target specimen.

Experiments use EMPIAR-10022 and an in-house α-synuclein dataset, under both standard and limited-annotation settings. The authors report much higher complete-instance recovery than the particle-picking baselines, and ablations examine the effects of synthetic pretraining, Fréchet supervision, and the architectural changes.

Critical strengths
  1. Predicting whole filament instances leads to large empirical gains.

CryoTracer treats filament continuity as part of the prediction target, rather than leaving it to a post-processing step that links local detections. This yields much higher complete-instance recovery than conventional filament-picking pipelines. The gain matters in practice: keeping filament identity intact helps downstream structural analysis and makes polymorph-specific processing easier.

  1. Ablations support the main design choices.

The ablations compare synthetic pretraining with real-only training, test alternative geometric losses, and vary the architecture. Synthetic initialization helps most when real annotations are scarce, and Fréchet supervision is important for recovering complete trajectories. Together, these results give reasonable support for the key components of the method.

Critical weaknesses
  1. Limited methodological novelty, and no comparison with related structured curve prediction methods.

Most of CryoTracer’s components are established techniques: transformer-based detection, polyline representations, deformable attention, Hungarian matching, and the Fréchet distance. The contribution is mainly in combining and adapting them for cryo-EM filament tracing.

This integration is useful in practice, but the experiments compare almost exclusively against particle-picking and detect-then-link pipelines. The paper does not compare its architecture with other learnable centerline or structured curve prediction methods. Without such baselines, it is unclear whether the gains come from the general move to direct polyline prediction or from the specific design choices in CryoTracer.

For an ICLR submission, the paper would be stronger if it showed a technical contribution that goes beyond integrating existing components for a specific domain.

  1. Generalization and behavior in difficult tracing cases are not well validated.

The evaluation uses only two real datasets, and both rely on specimen-specific fine-tuning. The α-synuclein data come from several acquisition sessions in the same lab, but the evaluation is not split by session. The results therefore do not show whether the method generalizes to unseen specimens, new acquisition sessions, or substantially different imaging conditions.

The primary metric, mF1, matches predictions to ground truth using the Fréchet distance, which is closely tied to the training objective. The paper also reports spatial coverage and Box IoU, but it does not analyze typical tracing failures: fragmentation, wrong merges at crossings, endpoint errors, and breaks across low-contrast gaps.

This gap is especially relevant because handling crossings and preserving continuity are the main motivations for the approach. A targeted analysis of these cases would show whether the reported gains translate into more reliable instance recovery where it matters most.

Questions for authors

Q1. How does CryoTracer differ technically from existing structured curve prediction methods? What distinguishes the architecture from prior polyline-based detectors, centerline predictors, and curve-based transformers? Comparisons against such trainable methods would make the specific methodological contribution much clearer.

Q2. How well does CryoTracer generalize to unseen specimens and acquisition sessions? All results use specimen-specific fine-tuning, and the α-synuclein evaluation is not split by session. Could the authors report leave-one-session-out results, or transfer results across specimens?

Q3. How does the method perform in the hardest filament configurations? Could the authors stratify results by crossing density, gap length, or curvature, and report fragmentation rate, incorrect merges, and endpoint error? These would test the claimed benefit of predicting whole trajectories more directly than mF1.

Q4. How much do mismatched training and adaptation protocols affect the comparison? The supplementary material notes that hyperparameter search and compute budgets differ across methods. RELION Amyloid is also used as a pretrained checkpoint without fine-tuning, while CryoTracer is adapted to each dataset. Could the authors discuss how these differences affect the results, or add comparisons under more closely matched conditions, for example with RELION Amyloid fine-tuned on the same annotations?

Q5. Do the gains hold under metrics that are independent of the training objective? mF1 relies on Fréchet-distance matching, which is also the training loss. Could the authors add an instance-level evaluation that does not depend on this criterion, such as an independent check of filament identity, to confirm the improvement is not specific to the chosen matching rule?

Reviewer_czNHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper the authors proposed a deep learning-based method for filament tracing in cryo-electron microscopy images. The model was trained on synthetic and real cryo-em data. Overall the work appears to be technically sound. But I also have several concerns. First, the paper is rather vague in presenting its detection model and synthetic training data generation. The lack of detailed information makes it difficult to assess the work. Second, experimental validation of the proposed model is limited. Third, the paper is rather specialized in its subject and may not be a good fit for ICLR. It is mostly a structural biology and image processing paper rather than a representation learning paper. Overall, the technical contribution of the paper seems limited.

Critical strengths

The paper presents a filament tracing method based on training of a detection model using synthetic and real cryo-em data. This provides better performance than previous methods that rely on postprocessing.

Critical weaknesses
  1. The paper is rather vague in presenting its detection model. Figure 1 does not provide adequate information about the model. Similarly, very limited information is provided on how the synthetic training data is generated. For example, how is background noise generated? What image formation model is used in generating the cryo-em images? The lack of detailed information makes it difficult to assess the work.

  2. Experimental validation of the proposed model is limited. Table 1 does not include results using RELION amyloid.

  3. The paper is mostly a structural biology and image processing paper rather than a representation learning paper.

  4. I assume the authors use LaTeX in preparing the manuscript. Please enclose the reference citations embedded in the text in parentheses.

Questions for authors
  1. Can RELION amyloid be used on EMPIAR-10022? Table 1 does not include comparison with this method.

  2. A variety of methods have been developed for curving tracing using deep learning models (e.g., DeepLSD, LINEA etc). Can the authors include a brief overview of such methods in "RELATED WORK".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.