- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper tries to study whether interval widths from a probabilistic imputer could be used for feature localization, independent-window monitoring, and individual error ranking. Then it narrows down the contribution to an evaluation protocol, telling people not to assume that interval widths' success on the task of feature localization does not mean the same for the other two tasks. To evaluate the idea, the main experiment uses a frozen flow-based imputer on PhysioNet 2012, with additional supervised error-ranking experiments on NIH-DEP. The author also validates the results through sensitivity analyses on parametes such as various patient pairings and Monte Carlo random streams. For the 3 tasks, the results show that the good performance on feature location would disappear after removing the shared randomness. Weak performance is observed in independent-window monitoring, and the additional value of interval widths would vanish after using a stronger probability-based baseline in individual error ranking. From the disclosed limitation, we know the findings are limited by the weak imputer and evaluation settings.
Critical strengths
The empirical validation is good. The paper conducts sensitivity analysis and negative controls to further evaluate how components like patient pairing, Monte Carlo randomness, and window size would impact the performance.
Critical weaknesses
- Motivation is not straightforward to me. The paper tries to define an evaluation protocol to prevent us from making wrong assumptions that do not directly apply interval width to independent-window monitoring or individual error ranking based only on the results of its performance on feature localization. But this does not make sense, as I do not think people would naturally link the success in one case scenario to its scalability to other scenarios with different target and test units.
- The weak imputer being used is not strong enough to help us to identify whether the failure in some tasks is from interval width or from the poor estimation of the imputer. I don't know why the author did not test the idea with stronger or multiple probabilistic imputers before drawing broader conclusions.
Questions for authors
-
For the motivation part or the contribution, I am just wondering whether the author identifies prior work that actually treats success on one of these tasks as evidence for success on the others? From my search through prior work, interval width or uncertainty estimation has been researched deeply for the 3 tasks separately, and I do not see any paper justifying that their evidence on the success of one task could be transferable to other tasks.
-
For the experiment, I would suggest the authors test strong imputers to help us identify whether the value of interval width could be enhanced or removed by a stronger imputer.
-
I did not find bibliography entries for the PhysioNet 2012 and 2019 datasets, as well as for the used Flow matching. It would be good to add a reference to each of them.