- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Rather than testing a hypothesis in order to solely reject or accept it, the authors suggest to test more meaningful aspects of the hypothesis such as whether it is unit specific or shared across the population. They suggest to test these aspects through a framework called NxN E-valuation. They evaluate how well each unit’s predicted restriction fits other unit’s outcomes and derive an NxN matrix. This matrix, combined with other statistical tests, determines whether predictions are unit-specific or shared and whether the hypotheses are meaningful and sufficiently widespread.
Critical strengths
The problem the authors aim to solve is important and well-motivated. Instead of testing a hypothesis in order to reject or accept it, they test whether the hypothesis is meaningful. They distinguish between unit-specific associations, shared population-level associations, base-rate effects, and subgroup-driven associations. This can further enable automated scientific discovery. The method is intuitive and useful. Creating the NxN matrix and using it to test the different aspects of the hypothesis is very appealing.
Critical weaknesses
-
As the authors acknowledge, the baselines that are considered (CRT, permutation test) do not target the same statistical problem, thus the comparison seems somewhat unfair.
-
Overall, the paper is well structured, but not written very clearly. Some explanations are difficult to follow, making the proposed methodology harder to understand.
-
The experiments are somewhat limited in their scope. All the experimental settings are based on recommendation systems. Including other types of hypotheses, such as continuous outcomes or other scientific associations would strengthen the paper.
-
It is unclear how to choose the coverage threshold and the parameter k, which determines how similar the restrictions must be in order to be considered shared. Although the authors provide sensitivity analyses, they do not suggest any data driven way to choose these parameters in order to achieve reliable diagnostic.
Questions for authors
None