- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes TomoDETR, a DETR-style detector for cryo-ET particle localization. Instead of predicting dense masks or heatmaps and then extracting centers, it regresses particle centers and classes directly from 64 grid-initialized queries in each overlapping 96³ window (Deformable DETR decoder, Hungarian matching). They add a second stage, the Cross-Window Query Refiner (CWQR). It is a sparse transformer over candidates coming from different windows, which refines centers and class logits before NMS. Validation is peformed on CZII, SHREC2021 and DeePiCt (VPP and defocus) benchmark datasets.
Critical strengths
The method is applied to an interesting problem, on several datasets.
Critical weaknesses
- The method is very hard to understand.
- The abstract, introduction and Fig. 1 never state what are the network outputs. Only Sec. 3.2 reveals that it regresses continuous centers from queries decoded over U-Net features.
- The notation is heavily overloaded. x denotes annotated, candidate and final centers. r denotes reference points, predicted centers and the graph radius. R denotes class radius and recall. P denotes precision and window width. β denotes the F-score parameter, a protein class and a focal-loss exponent. g denotes window geometry and the prediction heads.
- Several quantities are undefined or only defined in the appendix: g^win_i, E_q, ϕ_g, ϕ_w (Eq. 4), z and Ω (Eq. 6). The value r_g = 9.4614 is never justified.
- The baseline comparison may not be fair, and the gains outside CZII are not convincing.
- The CZII calibration set (40 tomograms) is larger than the training set (7). SHREC2021 uses 1 selection and 1 test tomogram, and DeePiCt uses 2 calibration and 2 test tomograms per split. The clearest gains appear on CZII, the dataset with by far the largest selection and test budget.
- On DeePiCt, the margins over DeepETPicker (+0.87 on VPP, +1.63 on defocus) fall within the standard deviations across splits, and no paired comparison is given.
- On SHREC2021, TomoDETR without CWQR (86.99) is below DeepETPicker (92.06), and the evaluation relies on a single test tomogram.
- External baselines are single runs. It is not stated whether they received the same tuning effort or the same select-then-retrain protocol (settings chosen on tomogram 8, retrained on 0–8).
- Limited methodological novelty. The first stage is a standard 3D Deformable DETR (Hungarian matching, L1+CE losses, deformable attention, auxiliary losses) with grid-initialized reference points. Point-based set prediction with Hungarian matching exists in other domains (e.g., crowd localization), and CWQR is conceptually close to learned NMS and relation-based refinement. The mismatch between confidence and localization quality is also known in object detection. The contribution is mainly an adaptation to cryo-ET plus a second-stage refiner, so its value depends on the empirical evidence, which is strongest on CZII only (with a lot of extra data for finetuning).
Questions for authors
- Did the baselines receive the same hyperparameters tuning and the same select-then-retrain protocol? Can you report paired per-split differences against DeepETPicker on DeePiCt?
- How was r_g = 9.4614 chosen? Please define g^win_i, z, Ω, E_q, ϕ_g and ϕ_w precisely, and state whether decoder updates are absolute or relative.
- Why is full CWQR 72.43 in Table 9 but 72.92 in Tables 1 and 2?
- Does the 1.16 GPU•h training cost include generating the cached candidates used to train CWQR?
5 comments
Response to Reviewer XtB1 (2/5)
W2 + Q1. Is the baseline comparison fair, and do the gains hold beyond CZII?
Reviewer concern. The baseline comparison may not be fair, and the gains outside CZII are not convincing.
The CZII calibration set (40 tomograms) is larger than the training set (7). SHREC2021 uses 1 selection and 1 test tomogram, and DeePiCt uses 2 calibration and 2 test tomograms per split. The clearest gains appear on CZII, the dataset with by far the largest selection and test budget.
On DeePiCt, the margins over DeepETPicker (+0.87 on VPP, +1.63 on defocus) fall within the standard deviations across splits, and no paired comparison is given.
On SHREC2021, TomoDETR without CWQR (86.99) is below DeepETPicker (92.06), and the evaluation relies on a single test tomogram.
External baselines are single runs. It is not stated whether they received the same tuning effort or the same select-then-retrain protocol (settings chosen on tomogram 8, retrained on 0–8).
Did the baselines receive the same hyperparameters tuning and the same select-then-retrain protocol? Can you report paired per-split differences against DeepETPicker on DeePiCt?
Response.
We appreciate these detailed questions on fairness and generality.
Tuning and protocol. All methods follow the same protocol, and each baseline was tuned on at least as much validation data as TomoDETR.
- Each external baseline uses its official code and the recipe from its paper or official example.
- We tuned each baseline on validation data with the selection mechanisms its code exposes (checkpoint or epoch, thresholds, cluster sizes, and post-processing settings; Appendix D.4, Table 21).
- No method, including TomoDETR, is retrained on tomograms 0–8: all methods train on 0–7, select on 8, and test on 9. The submission's mention of retraining was a description error and is now corrected (Sec. 4.1).
Single runs. We retrained all external baselines with three seeds (DeepFinder: two, as one run produced no predictions). On CZII, TomoDETR (72.92 ± 0.45) leads the strongest baseline, OCTOPI (66.81 ± 0.55), by 6.1 points. Table 1 and Table 5 now report three-seed means.
Selection set on CZII. The 40 selection tomograms (Val40) never update network weights; they only select checkpoints and thresholds. TomoDETR needs only a few of them (weighted F4 on the 364 test tomograms; mean ± SD over the three training seeds, with the N tomograms drawn at random from Val40):
| Selection tomograms N | 1 | 3 | 5 | 10 | 20 | 30 | 40 |
|---|---|---|---|---|---|---|---|
| TomoDETR | 68.58 ± 2.60 | 70.94 ± 0.85 | 72.03 ± 0.09 | 71.87 ± 0.70 | 72.52 ± 0.53 | 72.34 ± 0.60 | 72.92 ± 0.45 |
Five selection tomograms already give 72.03. Even with a single tomogram, the mean remains above the strongest baseline (OCTOPI, 66.81 ± 0.55).
DeePiCt, paired comparison. On DeePiCt, our claim is competitive rather than superior performance (Sec. 4.2). We added a per-test-tomogram comparison with DeepETPicker (Appendix D.3, Table 20). The differences (TomoDETR − DeepETPicker), two test tomograms per split, are:
- VPP: +0.70 and +0.37 (split 0), +0.01 and +1.23 (split 1), +2.95 and +2.33 (split 2). All six are positive, with a mean of +1.27.
- Defocus: +2.51 and −0.09 (split 0), +0.97 and −2.58 (split 1), +2.96 and +2.95 (split 2). Four of six are positive, with a mean of +1.12.
SHREC2021. To move beyond a single test tomogram, we added two further test tomograms (Appendix D.3, Table 19). TomoDETR has the highest F1 on all three: 93.08, 93.23, and 65.09, against the best baselines' 92.06, 92.69, and 64.28. We agree that TomoDETR without CWQR is below DeepETPicker here. However, CWQR is part of the method, and SHREC2021 shows its largest gain (+6.09).
Response to Reviewer XtB1 (3/5)
W3. Is the contribution more than an adaptation of Deformable DETR plus a second-stage refiner?
Reviewer concern. Limited methodological novelty.
The first stage is a standard 3D Deformable DETR (Hungarian matching, L1+CE losses, deformable attention, auxiliary losses) with grid-initialized reference points. Point-based set prediction with Hungarian matching exists in other domains (e.g., crowd localization), and CWQR is conceptually close to learned NMS and relation-based refinement. The mismatch between confidence and localization quality is also known in object detection. The contribution is mainly an adaptation to cryo-ET plus a second-stage refiner, so its value depends on the empirical evidence, which is strongest on CZII only (with a lot of extra data for finetuning).
Response.
We appreciate this candid assessment and agree that the first stage deliberately builds on Deformable DETR. We see the contribution in three points, each backed by controlled evidence.
- Point-set formulation for tomograms. Point-based set prediction exists for 2D crowd localization (P2PNet). To our knowledge, however, it has not been used for supervised particle localization in reconstructed tomograms, where dense prediction with center extraction is standard. Table 1(b) compares the two formulations under a shared backbone and training budget.
- Cross-window refinement, not single-image suppression. Learned NMS and relation networks operate on detections from one image under one context. CWQR instead jointly refines the coordinates and class scores of repeated observations of the same particle from different windows, under a single tomogram-level one-to-one assignment. Table 2 compares it with fixed fusion rules and with a parameter-matched self-only refiner.
- Confidence fails among repeated observations. The confidence–localization mismatch is known in detection (IoU-Net). Our finding concerns repeated observations of the same particle: the highest-confidence observation is the best-localized one only 16.72% of the time, the same as a random draw (16.53%) (Sec. 4.3). This motivates aggregating information across windows rather than selecting by confidence.
CWQR improves the local detector in all four settings, not only on CZII: +2.85 on CZII, +6.09 on SHREC2021, +2.18 on DeePiCt VPP, and +2.58 on DeePiCt defocus. On CZII, only the seven training tomograms update model weights; the 40 selection tomograms only select checkpoints and thresholds (see W2). We now cite P2PNet and IoU-Net in the related work (Sec. 2).
Response to Reviewer XtB1 (1/5)
W1 + Q2. Can the outputs, notation, and definitions be stated precisely, and how are r_g and the decoder updates defined?
Reviewer concern. The method is very hard to understand.
Response.
We are grateful for this careful reading and agree that the presentation needed to be clearer. We revised Sec. 3 as follows.
Network outputs. The abstract now states that the network outputs a set of continuous 3D particle centers with class labels and confidence scores.
Notation. The overloaded symbols are renamed:
Definitions. Sec. 3.3 now defines every term of Eqs. 4 and 6. The exact formulas and dimensions are in Appendix A.1.
Graph radius ρ (previously r_g). ρ is estimated on the CZII training tomograms only. It is 1.1 times the 99th percentile of the distances between candidates from different windows that are assigned to the same particle, which gives 9.4614 voxels. SHREC2021 and DeePiCt use 10 voxels, which equals the SHREC2021 particle radius and the DeePiCt matching tolerance (Appendix A.1).
Decoder updates. The updates are relative. As in Deformable DETR, each layer adds a predicted offset to the previous layer's center in inverse-sigmoid space (Sec. 3.2).