- Scope of the paper to ICLR is unclear: the paper simply proposes a statistical post-processing of the liver segmentation probability maps yielding derivate metrics for two different purposes: spatial gating and diagnostic probing of lesion LI-RAD grade swaps. The core argument that derived representations from liver CT volume segmentation maps serve as effective proxy LI-RAD features remains unjustified due to limited success with both in-domain and out-of-domain datasets (c.f. section 4.3, section 4.4., Table 2 and section 4.5). Is the paper trying to glorify an unsuccessful feature representation? Or is it trying hard to push the proposed uncertainty metrics as reasonable alternatives? I am afraid it's the former.
- LI-RAD grade measurement includes size and threshold growth as important features. It is unclear why these features were left out in the current study.
- Different LI-RAD scales reveal important characteristics of different aspects. Which of the LI-RAD scales are considered in the paper? What is the sample-wise distribution of the volumes across these grades? Some LI-RAD grades indicate benign nature of the lesion while some others indicate malignant nature, and some others reveal Hepatocellular Carcinoma status. Why should the proposed spatial gating and diagnostic probes serve uniformly across these different LI-RAD grades? A more detailed analysis of grade-wise flips, and grade-wise spatial gating results might give more insights into the proposal. This analysis however is missing.
- How APHE, non-peripheral washout and Capsule are measured in the current setup as LI-RAD features remains unclear. The discussion in Section 3.2 does not divulge details about explicit feature vector construction. In short: how are the three coordinates of f computed using the mask threshold sweep procedure?
- For a 3D feature vector, wouldn't a non-linear classification boundary be more reasonable than a linear classifier (despite Figure 4)? While the linear assumption might provide a nice closed form expression for the stay probability, it might be a potential cause for the non-uniform stability results across the LOW, MEDIUM and HIGH regimes. This is not explored further in the paper.
- Empirical comparisons are limited to ablated designs of the same scheme. Meritorious comparisons with existing counterfactual techniques and other uncertain quantification techniques are possible but dismissed as irrelevant in the Introduction and related work sections. This is possibly the major setback to the current form of the paper, given the poor tier-wise results discussed in Section 4.4 and non-transferable mechanism utility on an out-of-domain dataset shown in Section 4.5.
- Perhaps the final devastating blow to the proposal in the paper is revealed in Section 4.5. Neither the spatial gating nor the diagnostic probe utility are transferable to an unseen-domain data. The paper pitifully succumbs to the serious issues discussed in section 4.5. What does the paper want to argue on the out-of-domain analysis made on the LiTS data in section 4.5? Since concerns of Ovadia et al. 2018 are already well-established, wouldn't a reasonable argument be to alleviate such concerns using a suitable proposal than to just conform to those concerns?
Writing and unspecified details:
a. The paper's writing appears quite incomprehensible to a human reader, even after multiple readings.
b. What are the four geometric transformations considered in T used in computing PTTA?
c. What are the distance field Φ and vessel suppression map V? Why are they the only indicators of anatomical prior? Citations missing for this use.
d. The discussion in section 3.2 is based on a posterior P. What is this P? Is it the predicted segmentation mask probability obtained from
nnUnet without the test time augmentations? Also the discussion in section 3.2. involves a term called "phase". What is this?
e. Why is Δμ based on the ranges >0.25 and ≥0.75? What happens if these ranges are changed?
f. Why are the tiers LOW, MEDIUM and HIGH based on the specific ranges of Pflip? What happens if these ranges are changed?
Arbitrary experimental settings used in spatial gate and diagnostic probe construction:
i) Lines 213--214 indicate an additional experimental procedure in addition to eqn. (3). No ablations provided on this extra step.
ii) Arbitrary diagonals added to Σseg(u) in eqn. (5) and in lines 247--248, without a discussion.
iii) σCT= 10HU? Why?
iv)Why is the base noise considered to be (5,8,6) HUs?
v) Though α=1 is emphasized in section 4.2, an elaborate discussion on the tiered stability results in section 4.4 uses different values of α to discuss different implications. Is α important or not?
vi) Filtering schemes in Section 4.1. and 4.5 seem to be inconsistent.
General comments:
While the overall thought behind the proposed uncertainty metrics is interesting, everything would have made more sense if the results from nnUnet on liver tumor segmentation was comprehensive. However as indicated in the paper, results on WAW-TACE and LiTS show around 0.56 or 0.68 Dice. In presence of such poor quantifications from a trained model, goodness measurement of an add-on metric like the ones proposed in the paper will be of limited value. While research to bring improvements in liver tumor segmentation are still ongoing, the paper could have chosen a similar problem on an organ where models show impressive behavior (e.g. kidney, brain, etc).
Further the primary hypothesis that LI-RAD can be proxied using just three features (with unspecified feature creation steps) should be rigorously tested. Including other LI-RAD characteristics should be looked into.
To make any proposed uncertainty metric useful for LI-RAD scales, differential analysis should be carried out across different severity scales of LI-RAD grades.
I would urge the authors to improve their work based on these comments.