acceptodds
Under review as a conference paper at ICLR 2027

The microscope is the mask: privileged views and labels from a cryo-ET forward model

Abstract

In this work, we use simulated data to train a model for protein annotation in crowded cryo-electron tomography volumes reconstructed from images collected at limited tilt angles and severely corrupted by the measurement operator. Firstly, we leverage the corruptions imposed by the forward model to generate domain-specific augmented views of the exact same scene for an invariance objective integrated into the LeJEPA self-supervised training framework. Secondly, we use additional information from the simulation pipeline such as the positions and identities of proteins in the simulated volumes to inform the architecture of the model and the loss function, so that semantic information is localised at protein positions in the dense feature volume. The resulting model, CARNIVAL, is evaluated without finetuning on classification and detection tasks on a multi-protein benchmark dataset and two in-situ datasets spanning three processing types. We show that CARNIVAL outperforms a state-of-the-art model trained using a contrastive objective on simulated data with a single labelled protein per input block, rather than dense supervision over multiple proteins in the scene, in both tasks on the benchmark dataset, matches or outperforms it in in-situ classification, and computes feature volumes seven times faster. A nine-arm ablation across the five evaluation settings decomposes the mechanism, showing that scene-level supervision is essential and pointing to how the views interact as the source of further gains.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Cryo-electron tomography, self-supervised learning, simulation-based training, forward model, sim-to-real transfer, domain randomization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MixSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CARNIVAL, a simulation-based representation learning framework for protein classification and localization in cryo-ET tomograms. The framework constructs paired views by varying imaging conditions, background, and protein orientations, and combines view consistency with protein identity supervision, distributional regularization, and auxiliary segmentation. Trained only on simulated data, the model is evaluated without finetuning on experimental Phantom and in-cell datasets. Compared with TomoTwin, it improves classification and detection on Phantom, improves classification on C. reinhardtii, and extracts feature volumes faster, but performs worse in detection on five of seven in-cell tomograms.

Critical strengths
  1. Evaluation on experimental data. The authors evaluate both classification and detection on a controlled Phantom sample and two cellular systems, covering different acquisition and processing conditions. These experiments provide valuable evidence about transfer to experimental data and reveal limitations that would not be apparent from synthetic evaluation alone.

  2. Detailed data-generation procedure. The scene construction, relationships between the six views, and ranges of imaging and reconstruction parameters are described in useful detail. This makes the proposed data-generation strategy understandable and facilitates its reproduction.

Critical weaknesses
  1. The gains on Phantom do not establish improved generalization to the intended in situ application. A central motivation for cryo-ET is to identify macromolecules within their native cellular context, including their spatial organization and associations with membranes, filaments, organelles, and other complexes. Performance in this environment is therefore particularly important for assessing the practical value of the proposed annotation framework. However, CARNIVAL underperforms TomoTwin in detection on five of seven in-cell tomograms, despite improving classification in some settings. Classification at known particle positions does not resolve this limitation, since it bypasses the task of locating targets within the cellular background.

    The Phantom benchmark provides valuable experimental validation, but it tests a different setting: purified particles are mixed with cellular lysate to create a controlled sample. Although this captures real imaging artifacts and biological background, it does not preserve the full native organization of an intact cellular environment. Its construction also shares some characteristics with the proposed simulated scenes, which place proteins among membranes and small-protein backgrounds. A plausible alternative explanation is therefore that the new training distribution is better matched to Phantom, rather than that the proposed training framework learns representations that generalize better to native cellular environments. The combination of improved Phantom detection and weaker in-cell detection makes this distinction consequential, although it does not establish the cause.

    The current experiments do not adequately resolve this alternative explanation. TomoTwin uses an existing pretrained checkpoint, so both the training data and learning framework differ between methods. Retraining TomoTwin on data from the proposed simulator, while retaining its original objective, would help isolate the effect of the training data. Moreover, A1 matches the reference recipe on Phantom classification and outperforms it on in-cell classification, leaving the specific benefit of physics-based view pairing unclear. Thus, the results demonstrate useful transfer to experimental data, but do not sufficiently establish that the proposed mechanism improves generalization to the paper’s central application. Comparisons with recent related methods, such as CryoSiam and ProPicker, are also needed to assess its added value beyond TomoTwin.

  2. The central training procedure is incompletely specified, and the contribution is difficult to assess. The paper’s main methodological contribution concerns how simulated views and labels are used for representation learning. Nevertheless, the exact view-consistency objective, the pairing and weighting of the six views, the segmentation loss, and the loss coefficients are not fully specified, including in the appendix. These are central components of the proposed method, rather than incidental implementation details, and their omission hinders reproduction and assessment.

    The paper also does not clearly delineate the novel components relative to the adopted backbone, LeJEPA, and previous simulation-based approaches. In particular, the narrative emphasizes physically grounded view construction, whereas the strongest ablation evidence supports scene-level identity supervision. A clearer statement of the contributions, together with evidence corresponding to each contribution, is needed to establish what methodological advance the paper demonstrates.

Questions for authors
  1. Could the authors retrain TomoTwin on data generated by their simulator, while retaining its original training objective, and evaluate it on both Phantom and in-cell datasets? This would help determine whether the Phantom gains primarily reflect a better-matched training distribution rather than improved transferability of the proposed learning framework.

  2. Given that in situ annotation is a central motivation, what evidence establishes an advantage for the proposed framework in native cellular environments, where detection underperforms TomoTwin on five of seven tomograms? In particular, A1 matches the reference recipe on Phantom classification and performs better on in-cell classification. Could the authors provide detection results for A1 to clarify the benefit of physics-based view pairing?

  3. Please specify the exact view-consistency objective, the pairing and weighting of the six views, the segmentation loss, and all loss coefficients. Please also clarify which components constitute the principal methodological contributions relative to LeJEPA and prior simulation-based approaches.

  4. Could the authors provide comparisons with recent related methods, particularly CryoSiam and ProPicker under comparable conditions, or explain any specific incompatibilities that prevent such comparisons?

5 comments
AuthorsOpenReview ↗

Answer to Reviewer MixS (1/3)

We thank the reviewer for the constructive assessment, and in particular for proposing the TomoTwin retraining experiment, which is now in the paper. We provide our answers below, with changes in the revised PDF marked in blue.

W1: "The gains on Phantom do not establish improved generalization to the intended in situ application. [...] CARNIVAL underperforms TomoTwin in detection on five of seven in-cell tomograms, despite improving classification in some settings [...]"

We agree, and the revision says so plainly (abstract, Sections 1, 4.2 and 5). After a correction of the C. reinhardtii ribosome annotations (Appendix D), TomoTwin leads detection on all seven of these in-cell volumes, not five. The new in-cell evidence is on raw reconstructions of the three C. reinhardtii tomograms, the input that TomoTwin's workflow prescribes: there CARNIVAL leads in classification (0.964 vs 0.917) and in detection on all three tomograms (by 0.10–0.17; Tables 1 and 3). We claim no general advantage across all in-cell datasets.

W1 (continued): "[...] A plausible alternative explanation is therefore that the new training distribution is better matched to Phantom, rather than that the proposed training framework learns representations that generalize better to native cellular environments. [...]"

We agree that the Phantom is not the native cellular setting, and that this alternative explanation had to be tested: we ran the TomoTwin retraining you propose, using CARNIVAL's data (Q1). We would add only that the Phantom is experimental data built as a benchmark for this annotation task, so we consider a result on it informative in its own right.

W1 (continued): "The current experiments do not adequately resolve this alternative explanation. [...] Retraining TomoTwin on data from the proposed simulator [...] would help isolate the effect of the training data. Moreover, A1 matches the reference recipe on Phantom classification and outperforms it on in-cell classification [...]. Comparisons with recent related methods, such as CryoSiam and ProPicker, are also needed [...]"

  • Retraining TomoTwin: done (Section 4.3; see Q1). The simulated data explains part of CARNIVAL's lead on the Phantom, but not all of it.
  • A1: we added detection results and a second seed of the reference (see Q2), and no longer claim a benefit of physics-based pairing over generic augmentation.
  • CryoSiam and ProPicker: see Q4.

W2: "The central training procedure is incompletely specified [...]. the exact view-consistency objective, the pairing and weighting of the six views, the segmentation loss, and the loss coefficients are not fully specified [...]"

Section 3.2 and Appendix A now specify them.

  • Views: three clean and three corrupted views of one simulated scene. The clean views play the role of LeJEPA's global views and the corrupted views that of the local views.
  • LviewL_{\text{view}}: the mean squared distance from the embedding of each clean view to that of every view, clean and corrupted, of the same protein, with all pairs weighted equally.
  • LsceneL_{\text{scene}}: the mean squared distance between embeddings of proteins of the same type at different positions.
  • SIGReg: applied to the embeddings of the unique protein types in each batch (4096 slices, 41 quadrature points).
  • LsegL_{\text{seg}}: a binary cross-entropy on every voxel of a protein channel and a membrane channel (positive class weighted by 1 and 2).
  • Weights: 1.0 for LsceneL_{\text{scene}}, 0.1 for SIGReg and 0.1 for LsegL_{\text{seg}}, relative to LviewL_{\text{view}}. They were set early in development and not tuned.
AuthorsOpenReview ↗

Answer to Reviewer MixS (2/3)

W2 (continued): "The paper also does not clearly delineate the novel components relative to the adopted backbone, LeJEPA, and previous simulation-based approaches. In particular, the narrative emphasizes physically grounded view construction, whereas the strongest ablation evidence supports scene-level identity supervision. [...]"

The contributions are stated clearly at the end of Section 1, each with its evidence.

  1. The simulator, which runs on the GPU inside the training loop and provides clean and corrupted versions of each scene with the position and identity of every protein. Evidence: its data improves TomoTwin on the Phantom and on raw C. reinhardtii (Section 4.3).
  2. The training methodology. Relative to LeJEPA, clean and corrupted views of one simulated scene take the place of the global and local views, and a new loss, Lscene\mathcal{L}_{\text{scene}}, uses the scene labels on a dense feature volume. Evidence: the ablations (Section 4.4).
  3. The model, compared with TomoTwin.

We agree that the evidence is strongest for scene-level identity supervision, and the revised narrative follows it. LsceneL_{\text{scene}} is needed in every setting in both tasks, LviewL_{\text{view}} helps on every dataset except DS-10001, and whether the forward model or generic augmentation should separate the views is left open.

Q1: "Could the authors retrain TomoTwin on data generated by their simulator, while retaining its original training objective, and evaluate it on both Phantom and in-cell datasets? [...]"

Yes, this is now Section 4.3. We trained TomoTwin with its released architecture, loss and configuration on data from our simulator: a fixed set of 800 training and 200 validation subvolumes for each of the 112 protein types, as in TomoTwin's own training set, so that only the training distribution changes. We report the epoch with the best validation F1F_1, as TomoTwin's paper does.

  • Phantom and raw C. reinhardtii, where CARNIVAL leads the released TomoTwin: the retrained model lies between the two. In classification it is 0.01–0.04 above the released model and 0.02–0.04 below CARNIVAL. In detection its mean is 0.03 above the released model's on each dataset, and CARNIVAL is ahead of it on all seven volumes, by 0.07–0.12.
  • So your hypothesis is partly right: the simulated data accounts for part of CARNIVAL's lead, but not all of it. The remainder is not the objective alone, since the two models also differ in architecture and CARNIVAL draws new scenes throughout training.
  • The gain is not specific to the Phantom: it also appears on raw C. reinhardtii, an in-cell dataset. The simulator's protein types are those of TomoTwin's own training set above 100 kDa.
  • Denoised C. reinhardtii and S. pombe: the simulated data does not improve TomoTwin. Classification is within 0.01 of the released model, except on DS-10000 (0.04 lower), and detection is lower on all seven tomograms, by 0.05–0.12.

Q2: "Given that in situ annotation is a central motivation, what evidence establishes an advantage for the proposed framework in native cellular environments, where detection underperforms TomoTwin on five of seven tomograms? [...] Could the authors provide detection results for A1 to clarify the benefit of physics-based view pairing?"

There is no general advantage across all tested in-cell datasets: CARNIVAL is ahead on raw C. reinhardtii, and behind in detection on denoised C. reinhardtii and on S. pombe (see W1).

Detection for A1 (now AgenericA_{\text{generic}}) is in Table 20, against two seeds of the reference at the same 10,000 steps. It is above both seeds on 7 of 10 in-cell volumes and between them on the other 3; on the Phantom it is below both on the 4 main volumes and level with them on 22 additional ones. With one run at 10,000 steps, we cannot show that physics-based pairing adds anything over generic augmentation of one simulated acquisition, or the reverse, and the paper leaves this open. What the ablations support is what both arms share: the simulator, LsceneL_{\text{scene}} and LviewL_{\text{view}}.

AuthorsOpenReview ↗

Answer to Reviewer MixS (3/3)

Q3: "Please specify the exact view-consistency objective, the pairing and weighting of the six views, the segmentation loss, and all loss coefficients. Please also clarify which components constitute the principal methodological contributions relative to LeJEPA and prior simulation-based approaches."

See answers to W2 above, and the contributions are stated at the end of Section 1 in the paper, together with their evidence.

Q4: "Could the authors provide comparisons with recent related methods, particularly CryoSiam and ProPicker under comparable conditions, or explain any specific incompatibilities that prevent such comparisons?"

We did not run these comparisons. In the time available, we gave priority to the retraining of TomoTwin.

  • ProPicker: It encodes its prompts with TomoTwin and picks with its own promptable segmentation network. Our comparison with TomoTwin therefore covers its prompt representation, but not its picking network.
  • CryoSiam: As far as we understand, it identifies particles through modules trained on top of its embeddings and not by matching references, so the reference-based protocol in our paper does not apply to it directly.
Reviewer_TQZUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper present CARNIVAL, a 3D encoder trained for both tasks : classification and detection, of particles in cryo-ET (3D view of inside a cryogenic cell). This architecture is trained on synthetic data only, for 112 types of particles, like a previous work (TomoTwin). From one arrangement of proteins (112 PDB structures) and membranes, three clean and three corrupted views are built. Views 1 and 2 share protein positions and orientations but differ in the small-protein background and in independent draws of the acquisition parameters. View 3 keeps view 1’s background and parameters but redraws the protein orientations. These views feed the LeJEPA invariance objective (“the microscope is the mask”: the missing wedge plays the role of the mask). The “privileged” information is the complete scene labels (position and identity of every protein), available only in simulation. It is used for an identity loss between distinct proteins of the same type (L_scene) and for an auxiliary segmentation loss. The model is compared with TomoTwin (supervised triplet loss, one protein per subvolume, 120 training proteins) on three public datasets:

Phantom: 121 tomograms, 5 protein types, filtered and denoised. Detection is run on only 2 tomograms.

  • S. pombe (DS-10000, DS-10001): 10 tomograms each, ribosome and FAS. Detection is run on 4.
  • C. reinhardtii (DS-10301): 3 denoised tomograms, ribosome and nucleosome.

Results:

  • Classification: better on Phantom (macro F1 0.90 vs 0.86 filtered, 0.88 vs 0.80 denoised) and on C. reinhardtii (0.82 vs 0.64, driven by ribosomes), on par on S. pombe.
  • Detection: better on the Phantom tomograms, but TomoTwin wins 5 of 7 in-cell tomogram
  • Ablation (nine arms, 10k steps, one seed): L_scene is essential (−0.08 to −0.16 macro F1 in 4 of 5 settings). Arm A1, in which both views come from the same corrupted volume and differ only by generic augmentations (intensity, low-pass, noise, unsharp masking), matches the reference recipe on Phantom and improves in-cell classification by 0.08–0.12.
Critical strengths
  • The GPU forward model provides clean/corrupted pairs and complete scene labels, which real data cannot give. The ablation shows the resulting scene-level identity loss clearly matters (A3 and A5 lose 0.06–0.16 macro F1 in four of five settings). It is a simple, generic idea that could transfer to other imaging domains with a good simulator.
  • The evaluation uses three public datasets covering different modalities and a protocol with no dataset-specific tuning. Unfavorable results are reported openly (TomoTwin is better in situ and on ribosomes), which makes the paper credible and useful. The ~7× speed-up is also practically relevant.
Critical weaknesses

The ablation does not support the headline mechanism, and the released model is not the best recipe in the paper. A1, where the two views share one corruption (no independent draws of the forward-model parameters), matches the reference R on Phantom (−0.015 and −0.002 macro F1) and improves it in cells by 0.08–0.12. At only 10k steps, A1 already exceeds the final CARNIVAL (~84k steps) on both S. pombe sets (+0.09, +0.14) and on C. reinhardtii (+0.03), though it is below CARNIVAL on Phantom filtered (−0.04). I appreciate this ablation, but it makes me question whether the central claim of the pipeline, invariance to independent draws of the physics, is what drives the results, or whether the simulator plus L_scene is sufficient.

Questions for authors
  • The score of CARNIVAL on ribosome for DS-10301 (0.67) is much lower than on the other datasets, though much higher than TomoTwin. Where does this come from ? A1 reaches 0.72 there (Table 16). Does that indicate that invariance to independently drawn acquisition parameters is not what helps?
  • The reported detection F4 is the best over the threshold sweep. How could use the model in practice ? What F4 is obtained with a threshold fixed a priori or no threshold ?
  • Could you define L_view precisely (which pairs among the 6 views, which distance, how SIGReg is applied) and give λ_s, λ_r, λ_g and how they were chosen?
  • Why was detection run on only 2 of 121 Phantom tomograms and 4 of 20 S. pombe tomograms?
  • What motivates stride 8 for CARNIVAL ? Do you have a benchmark for different strides ?
  • The introduction states supervised learning on real tomograms is not a route, but no supervised baseline is shown. How do supervised methods, including ones trained on synthetic data such as Template Learning, perform, particularly on nucleosomes?
  • “Designing Affine invariant neural network for Photometric robustness and Generalization” M. Messaoudi et. al. reports photometric-invariant results on the Phantom dataset. It could be mentionned in the related work.
2 comments
AuthorsOpenReview ↗

Answer to Reviewer TQZU (1/2)

We thank the reviewer for the careful reading. We provide our answers below, with changes in the revised PDF marked in blue.

W1: "The ablation does not support the headline mechanism, and the released model is not the best recipe in the paper. [...] it makes me question whether the central claim of the pipeline, invariance to independent draws of the physics, is what drives the results, or whether the simulator plus L_scene is sufficient."

We agree, and the revision no longer presents invariance to independent draws of the physics as what drives the results. The contributions are now stated clearly at the end of Section 1, and whether the forward model or generic augmentations separate the views is left open.

  • Is the simulator plus Lscene\mathcal{L}_{\text{scene}} sufficient? No. We added an ablation arm without $L_{\text{view}} $ and a second seed of the reference (Tables 4 and 20). Without LviewL_{\text{view}}, classification falls by 0.03–0.05 on the Phantom, raw C. reinhardtii and DS-10000, and detection is below both reference seeds on the 12 main volumes outside DS-10001. On DS-10001, acquired with a Volta phase plate, which the simulator does not model, the view loss does not help. Removing Lscene\mathcal{L}_{\text{scene}} costs more: 0.05–0.14 in all six classification settings, and detection is below both seeds on all 14 volumes.

  • A1 ( now AgenericA_{\text{generic}} ) keeps the simulator, LsceneL_{\text{scene}} and LviewL_{\text{view}}; only what separates the two views of a pair differs. Against the two reference seeds, it is within 0.02 in classification on the Phantom and C. reinhardtii, and 0.12 higher on both S. pombe datasets, through FAS. In detection (new, Table 20) it is above both seeds on 7 of 10 in-cell volumes and between them on the other 3; on the Phantom it is below both on the 4 main volumes and level with them on 22 additional ones. We therefore do not claim that either way of separating the views is better, and we present A1's result as a hypothesis to test (Sections 4.4 and 5).

  • Best recipe. The reported model is now the final checkpoint of the run, at its scheduled 100,000 steps (the submission used a checkpoint at 84,000). Compared with AgenericA_{\text{generic}} at 10,000 steps, it is ahead in classification on the Phantom and C. reinhardtii and behind on S. pombe (0.63 and 0.66 vs 0.71 and 0.81). In detection it is ahead on every group of volumes except denoised C. reinhardtii (0.28 vs 0.30). We have a single AgenericA_{\text{generic}} run at 10,000 steps, so we cannot say which recipe is better at full length.

Q1: "The score of CARNIVAL on ribosome for DS-10301 (0.67) is much lower than on the other datasets, though much higher than TomoTwin. Where does this come from?"

The cause was the annotations. In two of the three tomograms, the deposited ribosome coordinates are displaced from the ribosome densities by about 13 and 9 voxels. We corrected them with one rigid transformation per tomogram, fitted to the tomogram density alone: no model output enters the fit, and an independent segmentation on the portal agrees with it (Appendix D). Every DS-10301 result now uses the corrected coordinates, for every model.

  • Ribosome F1F_1 on the denoised tomograms is now 0.91 for CARNIVAL and 0.93 for TomoTwin (0.66 and 0.32 at the deposited coordinates). CARNIVAL's classification lead on these tomograms is therefore gone: TomoTwin is ahead by 0.01 in macro F1F_1 (Table 1).
  • A1's advantage there is gone too: 0.87, against 0.86 and 0.84 for the two reference seeds (Table 16).
  • On the raw reconstructions of the same tomograms, which we added in the revised paper, CARNIVAL's ribosome F1F_1 is 0.935 against TomoTwin's 0.848 (Table 17).

Q2: "The reported detection F4 is the best over the threshold sweep. How could use the model in practice ? What F4 is obtained with a threshold fixed a priori or no threshold ?"

The best F4F_4 over the sweep is an upper bound available to neither method at deployment (Section 4.2), applied identically to both. We report it because a threshold fixed a priori is arbitrary and could favour either method; the full precision-recall curves are in Figures 4–7. Therefore, to avoid subjectivity, we have not computed F4F_4 at a fixed threshold. In practice, as in TomoTwin's reference-based workflow, the user can choose the references and set the threshold by inspecting the picks.

AuthorsOpenReview ↗

Answer to Reviewer TQZU (2/2)

Q3: "Could you define L_view precisely (which pairs among the 6 views, which distance, how SIGReg is applied) and give λs\lambda_s, λr\lambda_r, λg\lambda_g and how they were chosen?"

Section 3.2 and Appendix A now give these.

  • The three clean views play the role of LeJEPA's global views and the three corrupted views that of the local views.
  • Lview\mathcal{L}_{\text{view}} is the mean squared distance from the embedding of each clean view to that of every view, clean and corrupted, of the same protein.
  • SIGReg (4096 slices, 41 quadrature points) is applied to the embeddings of the unique protein types in each batch.
  • The weights are λs=1.0\lambda_s = 1.0, λr=0.1\lambda_r = 0.1 and λg=0.1\lambda_g = 0.1. They were not tuned: we set them early in development and kept them fixed. The ablations that set each weight to zero are in Table 4.

Q4: "Why was detection run on only 2 of 121 Phantom tomograms and 4 of 20 S. pombe tomograms?"

Detection needs a feature volume from both methods for every tomogram, and TomoTwin's takes about 30 minutes for a Phantom volume and 4 to 5.5 hours for an S. pombe tomogram on six GPUs (Table 6), so we evaluated a subset.

  • S. pombe: the four tomograms are the two with the most FAS annotations in each dataset (Appendix D.2).
  • Phantom: TS_5_2 is one of only three tomograms on which all five types have enough annotations to be scored, and TS_43_9 is dominated by one type.
  • New: to test whether the Phantom result depends on this choice, we ran 11 more tomograms: ten drawn at random, nine of which can be scored, and the two others on which all five types can be scored. CARNIVAL is ahead on 19 of these 22 volumes (0.61 vs 0.55 on the filtered tomograms, 0.55 vs 0.48 on the denoised ones; Table 12).

Q5: "What motivates stride 8 for CARNIVAL ? Do you have a benchmark for different strides ?"

We have no benchmark over strides. Stride 8 was chosen by visual inspection of feature volumes, not from a classification or detection score: coarser strides leave visible seams at the block edges, and stride 4 needs eight times as many blocks (Appendix D.2), which we believe would be too slow for CARNIVAL to be usable. Every result and the timing comparison use stride 8.

Q6: "The introduction states supervised learning on real tomograms is not a route, but no supervised baseline is shown. How do supervised methods, including ones trained on synthetic data such as Template Learning, perform, particularly on nucleosomes?"

We did not run one. Supervised pickers are trained for specific target structures, while CARNIVAL, like TomoTwin, is a single model applied to every protein type without training per target. TomoTwin is therefore the like-for-like baseline. A picker trained for nucleosomes may well do better on them: both general models detect nucleosomes poorly (F4F_4 at most 0.28), and we draw no conclusion from that class.

Q7: "“Designing Affine invariant neural network for Photometric robustness and Generalization” M. Messaoudi et. al. reports photometric-invariant results on the Phantom dataset. It could be mentionned in the related work."

Thank you for the reference, it is now cited in Section 2.

Reviewer_LmH5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CARNIVAL, a self-supervised model for cryo-ET classification and detection. Building on previous method TomoTwin, CARNIVAL replaces TomoTwin’s triplet loss with a LeJEPA-based objective as well as two additional auxiliary losses, a scene loss and segmentation loss. To generate multiple views for the LeJEPA-based objective, CARNIVAL proposes to generate both clean and corrupted views of the same specimen, as well as paired views with the same “stuff” but different orientations and backgrounds. This method is compared against TomoTwin on cryo-ET classification and detection.

Critical strengths

This paper’s idea of using LeJEPA to do representation learning in conjunection with invariances is interesting.

Critical weaknesses

(W1) Clarity: This paper is missing critical details about the paper, which make it impossible to fully assess the paper or to reproduce it. First, the segmentation loss Lseg\mathcal{L}_\mathrm{seg} is never defined. Second, while the paper mentions that the LeJEPA is used to do self-supervised training, it’s not clear how the views generated by the simulator are used in the LeJEPA loss. In LeJEPA, the views are separated into global and local views, with the non-SIGReg loss pulling all views towards the mean of the global views. In CARNIVAL, it is not clear which views play the role of the global views. The tiebreaking procedure for the leave-on-out k-NN classifier is also not clear.

Second, the ablations section (Section 4.3) is extremely difficult to understand and should be rewritten. It’s only a text, and a lot of it, and it’s extremely hard to tell which numerical results correspond to what datasets and keep the mapping of alphanumeric tags (e.g. R, A1) to ablations straight. I think that Table 4 should be moved to the main paper, and organized so different types of ablations occupy different sections (e.g., all the “loss removed” ablations get their own section). Second, the non-descriptive identifiers (e.g. A1) should be removed, with each identifier replaced by a short text description of the ablation (e.g. baseline - scene loss). Third, making reference to specific cells in the Table may make it easier to understand.

(W2) Empirical results: The empirical support for this paper has a few weaknesses: first, on the C. reinhardtii dataset, the authors used denoised tomograms, creating an unfair comparison with TomoTwin, which they describe as expecting not denoised tomograms (Lines 267-269). Similarly, on the Phantom dataset, detection is only evaluated on filtered or denoised tomograms, which were not the intended input for TomoTwin. This is verified by the paper’s own experiments: in the denoised tomogram classification setting, TomoTwin’s macro F1 drops by 0.06 compared to CARNIVAL’s 0.02, and for hard proteins the drop is larger and as much as 0.17 in F1, compared to ~0.04-0.05 for CARNIVAL (β\beta-galactosidase). This hurts TomoTwin even more on detection because hard proteins count for twice as much.

In the only remaining fair comparison (S. Pombe), TomoTwin consistently outperforms CARNIVAL in detection. Additionally, the detection task is a more realistic assessment of practical cryo-ET usage than the classification task, which assumes known particle positions and that the ground truth labels of all neighbors are known. Even in the classification task, results are mixed for S. pombe, the only fair real dataset comparison, with TomoTwin winning on the defocus and CARNIVAL winning on the phase plate, calling into question the real-world effectiveness of CARNIVAL.

Additionally, the ablations are not trained for the same number of steps as CARNIVAL, and there’s no verification that any of the results or the ordering of ablations holds up at the full 84K steps. In particular, critical ablations such as A1 should be trained for longer to verify that the results hold up at the full training length and to rule out alternative reasons for better results such as fast convergence.

Questions for authors
  1. How is Lseg\mathcal{L}_\mathrm{seg} defined and computed?
  2. How is LeJEPA actually applied to the generated views? In LeJEPA terms, which views play the role of the global views, and which are local views?
  3. What is the tiebreaking procedure for the LOO k-NN classifier?
  4. How do detection results compare between CARNIVAL and TomoTwin on raw tomograms for Phantom and C. reinhardtii datasets, if available? How about classification for C. reinhardtii?
  5. How do the ablations compare at the full CARNIVAL training length?
2 comments
AuthorsOpenReview ↗

Answer to Reviewer LmH5 (1/2)

We thank the reviewer for the detailed comments on clarity and on the comparison with TomoTwin. We provide our answers below, with changes in the revised PDF marked in blue.

W1: "Clarity: [...]. First, the segmentation loss Lseg\mathcal{L}_{\text{seg}} is never defined."

It is now defined in Appendix A: a binary cross-entropy on every voxel of two output channels, against masks obtained by thresholding the simulated proteins and membranes, averaged over the six views. The segmentation head is optional and its output is used in no result in the paper.

W1 (continued): "[...] it’s not clear how the views generated by the simulator are used in the LeJEPA loss. [...] In CARNIVAL, it is not clear which views play the role of the global views."

Section 3.2 now states that the three clean views are the global views and the three corrupted views are the local views. Lview\mathcal{L}_{\text{view}} is the mean squared distance from the embedding of each clean view to that of every view, clean and corrupted, of the same protein. We can think of the clean views as being global views in Fourier space, while the corrupted views are local views in Fourier space, as they only contain a subset of the Fourier coefficients due to the missing wedge.

W1 (continued): "The tiebreaking procedure for the leave-one-out k-NN classifier is also not clear."

Appendix D.1 now gives it: each of the k=5k=5 neighbours casts one vote, and a tie goes to the first of the tied classes in a fixed (alphabetical) order. The in-cell datasets have two classes, so no tie can occur there.

W1 (continued): "Second, the ablations section (Section 4.3) is extremely difficult to understand and should be rewritten. [...]"

Your suggestions for rewriting it have been very helpful. This is now Section 4.4. Table 4 now shows the ablation arms grouped by type of ablation, descriptive names in place of the tags (for example Ano-sceneA_{\text{no-scene}}), and one paragraph of text per group drawing conclusions. We also added a second seed of the reference, so an arm is read as different only where it lies outside both seeds. Moreover, we added detection results for three ablation arms in Table 20.

W2: "Empirical results: [...] on the C. reinhardtii dataset, the authors used denoised tomograms, creating an unfair comparison with TomoTwin [...]"

We agree that this comparison needed raw input. While we had initially only used readily available tomograms from the CZI CryoET Data Portal (such as the denoised C. reinhardtii ones), we have now reconstructed the three tomograms from the deposited tilt series by weighted back-projection, with no CTF correction and no denoising, which is the input that TomoTwin's workflow prescribes. On these, CARNIVAL leads in classification (macro F1F_1 0.964 vs 0.917) and in detection on all three tomograms (0.44, 0.28, 0.40 vs 0.27, 0.19, 0.30; Tables 1 and 3).

The numbers on the denoised tomograms have also changed. We found that the deposited ribosome coordinates are displaced from the densities in two of the three tomograms, and corrected them (Appendix D). At the corrected coordinates, CARNIVAL's classification lead of the submission (0.82 vs 0.64) is gone: TomoTwin is ahead by 0.01, and it leads detection on all three denoised tomograms. TomoTwin scores higher on the denoised tomograms than on the raw ones in both tasks.

W2 (continued): "Similarly, on the Phantom dataset, detection is only evaluated on filtered or denoised tomograms, which were not the intended input for TomoTwin. [...]"

Our description of the "filtered" Phantom tomograms as band-pass filtered was wrong and is now corrected: they are weighted back-projection reconstructions with CTF correction and no denoising. TomoTwin's documentation asks for tomograms without denoising or low-pass filtering and does not mention CTF correction, so we consider the filtered form within its documented input. The denoised form is outside it, as the paper says. On the filtered form, CARNIVAL leads in classification (0.89 vs 0.86) and in detection on 11 of 13 tomograms, 11 of which are new in the revision (Tables 2 and 12).

W2 (continued): "In the only remaining fair comparison (S. Pombe), TomoTwin consistently outperforms CARNIVAL in detection. Additionally, the detection task is a more realistic assessment of practical cryo-ET usage than the classification task [...]"

We agree on both points and say so in the abstract and in Sections 4 and 5. TomoTwin leads detection on all four S. pombe tomograms (by 0.01–0.09), and classification there is on par, with the sign of the difference depending on kk. Raw C. reinhardtii is now a second in-cell comparison on TomoTwin's documented input. The revision also states that classification at annotated positions does not predict the detection differences (Section 4.2). We keep classification because it isolates the embedding from the detection procedure.

AuthorsOpenReview ↗

Answer to Reviewer LmH5 (2/2)

W2 (continued): "Additionally, the ablations are not trained for the same number of steps as CARNIVAL [...]. In particular, critical ablations such as A1 should be trained for longer [...]"

A full-length run takes about a month on a six-GPU node, so we could not train the arms to 100,000 steps during the discussion period. We do not claim that the ordering at 10,000 steps holds at full length, and for A1 (now AgenericA_{\text{generic}}) we no longer claim that either way of making the views is better.

One observation on length: from 10,000 to 100,000 steps, the reference recipe gains little in S. pombe classification (0.57–0.59 to 0.63 on DS-10000), so longer training does not close its gap to AgenericA_{\text{generic}} there (0.71). In detection it gains more (0.56 to 0.65 on the main Phantom volumes), so we cannot rule out faster convergence as an explanation of AgenericA_{\text{generic}}'s detection results.

Q1: "How is Lseg\mathcal{L}_{\text{seg}} defined and computed?"

Answered under W1.

Q2: "How is LeJEPA actually applied to the generated views? In LeJEPA terms, which views play the role of the global views, and which are local views?"

The clean views are the global views and the corrupted views are the local views; see W1.

Q3: "What is the tiebreaking procedure for the LOO k-NN classifier?"

A tie goes to the first of the tied classes in a fixed order; see W1.

Q4: "How do detection results compare between CARNIVAL and TomoTwin on raw tomograms for Phantom and C. reinhardtii datasets, if available? How about classification for C. reinhardtii?"

  • C. reinhardtii: on raw reconstructions CARNIVAL leads in classification (0.964 vs 0.917) and in detection on all three tomograms; see W2.
  • Phantom: no reconstruction without CTF correction is deposited for this dataset. The filtered form has no denoising and is within TomoTwin's documented input; see W2.

Q5: "How do the ablations compare at the full CARNIVAL training length?"

We do not have the ablations at full length; see W2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes in the revised PDF

We thank the three reviewers for their detailed and constructive reviews. We have uploaded a revised PDF, with the changes marked in blue, and we answer each reviewer separately. This comment summarises what changed.

New experiments

  • TomoTwin retrained on our simulated data (Section 4.3), to separate the effect of the simulator from that of the rest of the method.
  • Raw C. reinhardtii tomograms, the input that TomoTwin's workflow prescribes, for both classification and detection (Tables 1 and 3).
  • Detection on 11 additional Phantom tomograms (Table 12).
  • Ablations: a second seed of the reference arm, an arm without Lview\mathcal{L}_{\text{view}}, and detection results for three ablation arms (Tables 4 and 20).

Corrections

  • Reported model. It is now the final checkpoint of the training run, at its scheduled 100,000 steps. The submission used a checkpoint at 84,000 steps. Every result of the CARNIVAL model is updated.
  • Ribosome annotations of the C. reinhardtii dataset. The deposited coordinates are displaced from the ribosome densities in two of the three tomograms. We corrected them (Appendix D) and re-evaluated every model. On the denoised tomograms, CARNIVAL's classification lead of the submission is gone and TomoTwin leads detection on all three.
  • Phantom "filtered" tomograms. They are CTF-corrected weighted back-projection reconstructions without denoising, not band-pass filtered as the submission stated.

Claims and presentation

  • The contributions are stated clearly at the end of Section 1, each with its evidence, and the abstract, Introduction and Discussion follow them. Compared with TomoTwin, CARNIVAL is ahead in both tasks on the Phantom and on raw C. reinhardtii, and behind in detection on the other in-cell data.
  • We no longer claim that separating the views through the forward model is what drives the results. This question is left open.
  • The ablations section is rewritten (Section 4.4), with a clearer Table 4.
  • The training objective and the evaluation protocol are specified in full (Section 3.2, Appendices A and D).
  • The timing comparison now covers every dataset and adds model size (Section 4.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.