- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents a dataset pruning method. First the difficulty of each sample is evaluated by AUM or variants. Each difficulty level (there are 50 or so such levels) is initially pruned at the same level depending on the budget. Then these parts are probed and a new pruning levels are constructed according to their notion of "excess fitting" by comparing the trained full model's accuracy and the checkpoint's accuracy.
Critical strengths
The paper develops some interesting ideas like excess fitting on dataset pruning that may be applicable in different context. The paper is generally well written.
Critical weaknesses
- First training a model to assess difficulty, then not stopping there are retraining full using parts of the model implies enormous computational complexity and defeats the purpose of dataset pruning. The idea of dataset pruning is that one already has a huge dataset and want to "quickly" get rid of irrelevant parts to speed up overall training. Assessing difficulty after full training (and actually doing this again for the probing subsets) does not seem to me an interesting problem. In the paper, the argument is that this is done "once" and then applied to other downstreams tasks but I do not find this convincing as every downstream tasks would have its own notion of easy and difficult. This is the major issue with the paper.
- The sentence "Existing difficulty scores are budget-agnostic: they reflect each example’s contribution to learning under full-data training, which may differ at the target budget." did not really make sense to me despite multiple times reading it and I do not understand the intention. This claim of existing algos are budget agnostic is also repeated multiple times... In fact, all algos work with some concept of a budget (e.g. in the form of a threshold in the simplest case, threshold low means budget high or threshold high means budget low). Also, in their simulations they have previous work under the umbrella of "budgeted method" which seems to contradict this claim.
- There has been other recent work on dataset pruning that the paper does not elaborate or compare to. The reference list seems limited in this context.
- The theorems seem to be thrown around for the sake of being theorems and do not establish the critical facts, e.g. why this scheme is supposed to be better than the others.
Questions for authors
No questions.
Dear Reviewers,
The authors have finished uploading their responses to reviewers, and the discussion phase is now underway. If you haven't already, please carefully review the authors' responses to understand their perspectives. Please engage in thoughtful and constructive discussions with the authors, and update your review or rating if necessary.
The discussion period lasts until November 18th AoE, which means you have less than five days left to interact with the authors. Please start engaging well before the period ends, so that the authors will have sufficient time to respond to any follow-up questions/comments.
Your active participation and contribution to the ongoing discussion are highly encouraged. Thank you very much for your contribution to ICLR.
Best regards,
AC