- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates when a single influence score is sufficient for data selection across multiple target capabilities. The authors first analyze the geometry of first-order capability improvements, showing that common selection rules coincide under a positive rank-one structure and deriving performance guarantees for near-rank-one cases. They then introduce replication cost and arbitrage gap to characterize whether a candidate provides additional first-order improvements beyond those achievable by the selected data, using LP duality to establish a local learning interpretation. Experiments on CV benchmarks, including RotatedMNIST, PACS, and OfficeHome, examine these theoretical findings. The results show that scalar influence can underperform random selection, while the predictive value of arbitrage gap is limited to local updates. Furthermore, first-order coverage helps identify poorly performing selection methods, whereas target-gradient distance better distinguishes stronger methods. Overall, the paper provides theoretical and empirical insights into the limitations of scalar influence-based data selection.
Critical strengths
The problem studied seems interesting. The experiments are extensive, with theory also provided.
Critical weaknesses
-
The paper is a bit hard to read and follow. For example, the abstract is written in an ambiguous way. Taking the second sentence, i.e., "At rank one with a positive direction, fixed prices, task-wise max, round-robin, capability-level BIDS and greedy gap maximization select the same data, and near rank one a fixed linear price keeps a guaranteed share of every capability’s best gain, tight up to a constant for two capabilities." as an example, I personally cannot understand those terms such as "positive direction", "two capabilities" mean in their contexts. There are many similar reading gaps, making a human reader feel frustrated and the paper hard to follow. A throughout pass could improve this paper's readability significantly.
-
The theoretical results look rather simple, and it's hard to decipher the connection with later experiments.
Questions for authors
A general recommendation is to do a careful sanity check and an overall proofread to polish and reorganize this paper so its presentation can be further improved.
What are the practical examples where the developed theory can be applied? How can the characterized region in Fig 1, with the detailed influence metrics, such as 0.565 vs.0.51 be used to improve data selection?