- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether internal organization of models (particularly, graph metrics that attempt to measure small-world organization) relates to capability emergence across scales and tasks. They evaluate these metrics and performance across several model families, scales, and task domains, and perform additional analyses such as interventions on identified networks to attempt to validate these results.
Critical strengths
The idea of relating the performance of the system to its internal functional organization using graph metrics is interesting, and could be useful if thoroughly validated.
Critical weaknesses
Although I find the idea somewhat interesting, I think the evidence in the paper is far for sufficient for establishing a meaningful relationship between small-world organization and task performance.
-
The main results in the paper are essentially correlational analyses — just observing that the metrics change in similar ways across scales with the tasks — on 4 data points per task, without even statistically quantifying the strength of those relationships.
-
These analyses may be heavily influenced by particular choices in performing the analyses. For example, because of the clipping in the SWP metric, the scores floor at 1 - 1/sqrt(2) and many of the plotted curves are at this floor before emergence. We therefore might miss important organizational changes below this floor (in a similar way to how Schaeffer et al. argue that discrete metrics create the appearance of emergence).
-
There are many other graph organization metrics (betweenness-centrality, degree distributions, etc.) that have been related to functional organization in networks (e.g., in neuroscience). The paper does not make any effort to show that SWP is uniquely related to accuracy relative to any of the other graph metrics that might have been considered.
-
Although intervention experiments could help to resolve these issues, the experiments used at present only show that the attribution method finds contributors to the performance; it doesn’t show that the small-world organization per se is the relevant factor (as the paper itself notes “The interventions do not isolate the causal contribution of small-world topology”).
-
etc.
Questions for authors
- Can you quantify the statistical significance of the relationship between the SWP metrics and performance, and show that floor effects, etc. aren't driving that relationship?
- Can you provide evidence that the small-world metrics are uniquely predictive of performances changes relative to other commonly used metrics of graph structure?
- Can you perform interventions that, again, selectively validate that small-world organization is the unique structural feature that determines performance?