- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors want to tackle that MRI's do not show the full patient trajectory and multiple onset of lesions could have explained the current state of the patient. In this sense it is hard to estimate Progression independent of relapse activity. The authors designed a framework based on aggregating possible lesion histories in a unified framework to improve upon the state-of-the-art.
Critical strengths
The authors tackle an important problem as pseudoprogress is hard to spot. Thinking about the stochastic nature of the problem is a valuable effort and the theoretical framework developed is extensive.
Critical weaknesses
Dataset issues:
- There are 37 pages in the paper and none of them show a single example from their datasets. No actual FLAIR images are shown whatsoever. Fig 1 uses drawn images and in Fig 2 I am not sure if these are even FLAIR images as v_pk looks like a T1w image. It would really help the reading flow to actually see what is predicted and how it is predicted, especially because they are working on a private dataset. Same with the formulas. The figures do not present well, how these formulas adapt the different quantities and how this merging works in principle.
- Furthermore, the public datasets are not well suited for this task as they only have 2 visits per patient. How do you delete an intermediate visit, when the dataset only has 2? Eligibility on the private dataset are patients with at least 3 visits. How do these results correlate?
- There is a publicly available longitudinal MS dataset with multiple visits given by Carass, Aaron, et al. "Longitudinal multiple sclerosis lesion segmentation: resource and challenge." NeuroImage 148 (2017): 77-102. albeit just 5 patients.
- A single private testset with 240 patients is inconclusive for any kind of generalization claims.
Motivation of the individual components:
- The authors failed to motivate any of their formulas in the entire paper. It is also not clear what they are doing differently from existing work and where they improve upon these works. There is no novelty statement and from the introduction it reads like they took a couple of components and stuck them together to improve upon the state-of-the-art in this application. Similar from the related work. Are you proposing a new model for the specific trajectories? Is the aggregation novel? Is it only the setup? This should be further specified?
- In any case many of the equations, like Eq 1 are just stated, however they it is never explained why this is a good choice. It is not even taken from any citation, nor is it derived from anything. Same with their theoretical setup. Some introductory words what the authors are actually trying to prove and accomplish would significantly enhance this paper. The figures are also very basic and do not explain the content well. Why should anyone use these formula and where are they applicable? The paper reads like a technical report without any insight into why and how any of these components are used and where the limitations lie.
Questions for authors
-
Can you clarify how the results on your public benchmark relate with the evaluation on the private testset, given that the scan numbers are vastly different.
-
Could you visualize the actual workflow based on images to get a better understanding on how your framework operates?
-
Can you add motivation for the different components and detail the degree of novelty of these components?