- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an auditing procedure—comprising three thresholds and three permutation tests—to determine whether transfer scores contain information specific to task pairs that cannot be explained solely by task effects, model effects, or noise from random seeds. The study evaluates various confounding factors and failure scenarios through synthetic experiments. Within a collection of seven base models, LogME, H-score, and N-LEEP passed the audit 11 times out of 84 trials.
Critical strengths
The paper distinguishes between general correlation and task-specific information, examining the respective influences of tasks, models, and noise. It provides actionable auditing procedures, thereby helping to raise the evidentiary standards for claims regarding transfer scores.
Critical weaknesses
The default audit process provides inadequate control over error rates. As shown in Table 5, the error pass rate ranges from 6.2% to 8.4% under conditions of heteroscedasticity—a phenomenon also significantly present in real-world medical datasets. Furthermore, the conditions for applicability require further clarification, particularly regarding dependencies within model families, the handling of missing data, and the thresholds used to manage noise.
Questions for authors
- Why is the condition ($ N \leq D $) sufficient to rule out all three categories of scoring methods?
- Do the findings obtained from the 11 successful audits still hold when using held-out models or tasks, conducting independent replications, or employing alternative ways of constructing outcome variables?