- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper claims to proffer an analysis of the relationship between predictive uncertainty and sampling density. While this is generally accepted in the community, the authors find that this is not the case. The authors claim that after pre-processing this relationship between sampling density and uncertainty does not hold. Further, the authors analyze an empirical upper limit of how much information the sampling can give about the predictive error, which is quiet low according to the authors.
Critical strengths
The biggest strength might be in going against the grain. The relationship between sampling density and prediction error is largely accepted in the community, with minor caveats. If there is little or no mutual information in the sampling schedule and the prediction error, this would raise a very important point with ramifications on active learning, design of experiments, etc.
Critical weaknesses
- The paper is not well written at all. I had to read it many times before understanding it and I still dont understand all of it. There are multiple levels to this a) Undefined Terms: For instance, the authors use the term NABU over 50 times in the article, but it is never defined. What is this NABU? I work in time series methods and have never heard of this. There are multiple such terms. b) Circular or self-referential writing: As an instance, on page 4: "An earlier draft called NABU “theory-grounded”; we no longer do...". An earlier version of what? The reviewer has access to this draft and this draft only. I have not seen any other versions. Why do I need to know this? Similar allusions to "earlier versions" are scattered all through the submission. Whilst I am raising a few of these issues, I feel that the draft is rife with such shortcomings.
Questions for authors
I recommend a re-write from scratch. My rationale are shared in the section of Critical Weaknesses.