- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper aims to explore how much of a model behaviour can be approximated by behaviour of its peers, specifically via convex-peer fit. Peer removal allows quantitative measure of which peer contributes the most.
Critical strengths
None
Critical weaknesses
- To claim approximation of model behaviour based on a single benchmark, MMLU Pro, is too weak a claim.
- I don't see why this is an interesting problem to begin with. What do we expect to do with the "coverage"? Downstream, the only thing I can imagine is perhaps how to best construct an ensemble of models, but there is no clear evidence that the given analysis will help with that. I don't see how this is related to feature attribution, which is listed as related work.
- I admit this is outside my area, but even then some parts were very difficult to read and understand.
Questions for authors
- What can we do better with this information? And do we have any evidence for that?