- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a method for automated revision support framework in histology slides for cell classification with post-hoc conformal prediction. Several baselines and datasets (mostly pathology with WildCam) and show moderate improvement over methods such as CytoFormer.
Critical strengths
The empirical results show some improvement on the task over multiple datasets. Although the experimental setup and evaluation metrics need more exposition instead of notation and large multi panel figures.
Critical weaknesses
- The paper mixes previous existing methods e.g. conformal prediction with the revision setting and is it unclear where the novel insight or contribution lie. Otherwise the paper is straightforward application of known/existing work.
- There is no validation of clinical utility. No reader study or reference to how revision helps in practice. It is unclear how such a framework could be integrated into clinical workflow or deployment.
- In general, much of the main text notation is only used once and can be simplified for readability. There are presentation issues with undefined notation and missing implementation and experimental setup details that make the paper hard to follow.
Questions for authors
- What general insights beyond this application?
- What is the scope of the novel contribution? what has been done before? Missing related works. Needs to be more demarcation between what is the contribution versus what is reusing existing or previous results. A explicit contribution list would help.
- Writing is obtuse and dense and littered with unnecessary notation that makes it difficult to understand the proposed method and implementation details. An algorithm box might help. Or at least add the same notation to Fig 2 i.e. z, W, U, beta. How is a and r formally defined? What is the bank and action space? What does audit mean here? Is b and h the same as B and H? Otherwise h is overloaded for harms and candidate proposal function. Subscript e and i are never defined. Is subscript e it same as superscript e?
- In Eq. 5, What is a support action? Where does 128 come from?
- How does the finite-sample assumption affect the audit procedure?
- What is clinical significance of the revision? is it practically useful? Is there any real user studies showing improvements from this automated/reference setup?
- It is unclear what the conformalizing procedure is over. Is f_i in Eq. 8 the same as Eq. 1? How is calibration score s defined in your implmentation. The threshold t is not formally defined. n is overloaded. How is alpha chosen in practice?
- What is the distinction between "fit" and "held"? So for Fig. 3d, how should interpret the AUC plot?
- Undefined jargon like "Private rich-router" and "Serial information-flow contract". How is information beyond the standard binary predictions analysis?
- Missing standard deviation and significance testing against baselines like CytoFormer.
- What is out of distribution (e.g. hospitals, demographics) performance of your method? Can you do cross dataset evaluation such as train on WILDS, test on Lizard?
- Seems like typo in Table 2 for the PanNuke row?