- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper present a self-supervised based anomaly detection on chest X-rays. The method itself leverages transformer based encoder-decoder architecture that works based on row and column multi-head attention. Moreover, the method uses a learnable timestep embedding added to the patch and positional embeddings. Anomaly scores follow the convention of reconstruction based approaches. The authors evaluate their method on X-ray benchmarks including COVIDx CXR-4 (96.51% AUROC), with generalisation assessment on the Kaggle pneumonia dataset without retraining (97.16% AUROC). The authors used variant baselines including supervised classifiers supplemented by ablation studies, quantitative results and method components such as Attention mechanism and time embedding.
Critical strengths
-
Regarding the experimental results, reported performance on two different datasets, including evaluation on pneumonia without retraining, is promising
-
The ablation results are informative mainly in Table 2 on the effect of method components including the attention mechanism, the skip connections and the time embedding. Also, outperforming global attention (96.51% vs. 92.34% AUROC) while reducing computational complexity, making a strong use case.
Critical weaknesses
-
I believe there is an inconsistency in the algorithm which makes me confused. In Algorithm 1, page 15, the network is trained to predict the clean image, i.e., (f_\theta(x_t,t)\approx x_0). However, at test time for inference, this prediction is directly used as (x_{t-1}) and passed to the network for multiple steps. This is different from conventional diffusion update, which builds the next noisy state using the predicted clean image and the noise schedule, so I wonder after the first iteration, inputs may drift from the expected noise distribution. Additionally, the reported noise schedule leaves almost no original image information at T = 1,000 steps that authors reported in implementation details. Can authors clarify this?!
-
I am also unsure about the deterministic claim in the abstract, as algorithm builds teh initial state using stochastic forward diffusion, so eliminating noise injection during reverse sampling does not make the overall reconstruction deterministic. Am I right?
-
The main weakness is the experimental evidence, as it lacks many recent self-supervised baselines for chest x-ray anomaly detection. It only covers supervised classifiers while many well-known reconstruction-based or diffusion-based anomaly detectors are missing. It is also hard to judge the claim of superiority especially because the proposed components are already well-known in the field.
-
Also, some reported metrics in Table 1 just don't add up under the binary-classification definitions. For example, a sensitivity of 77.43% cannot end up with an F1 score of 90.67%, and a similar inconsistency for the pneumonia CNN baseline (83.33% sensitivity and 96.15% F1).
-
There is also ambiguity about threshold selection. In Section 3.4, it is written that the detection threshold is selected by maximizing validation F1, whereas Section 4.1 describes using only normal validation images. Authors needs to clarify these inconsistencies. I also could not find the actual runtime or memory measurements, which is another weakness of this paper.
Questions for authors
-
Please see my comments above regarding clarification of the reverse process in Algorithm 1, and Equation 9. Please clarify the exact implemented update.Also comparison w.r.t. DDIM sampler would be beneficial. Besides, how sensitive the predictions and anomaly score are w.r.t. initial forward-diffusion noise realization?
-
Please also find my above comments in critical weaknesses on metric inconsistency and evaluation protocols which need clarification. There is lack of recent diffusion-based reconstruction baselines as I described above and compared baselines are limited to supervised classifiers only.