- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-
In the paper, the authors proposed Recycle Then Drop (RtD) for dynamic data pruning, in which, within each mini-batch, instead of dropping the pruned samples, they keep a portion of them and apply them with stronger augmentation (recycle set).
-
Their default pruning score is the ground-truth-class softmax probability.
-
They show compatability to several pruning methods.
-
Also, they proposed Amortized RtD - a version that enables to reuse the scores computed for samples for later use to reduce the training time.
-
Experiments conducted on image classification tasks (CIFAR-10/100 and ImageNet-1k), object detection and semantic segmentation on PASCAL-VOC.
-
The RtD is compared to multiple relevant baselines and across several benchmarks.
Critical strengths
- Broad comparison to baselines (both static and dynamic pruning) and different architectures. Adding the non-classification tasks (object detection and semantic segmentation) is useful.
- The method was tested with several pruning approaches.
Critical weaknesses
- It seems like the main results in the paper at Table 1 are based on unfair comparison between the RtD and the full-data baseline. While the RtD uses strong augmentation for the recycled samples (“Recycled samples additionally receive HorizontalFlip and RandAugment”), the baseline methods used only basic augmentation. Therefore, the claim that the RtD achieves higher accuracy than the full-data training is problematic (unless I miss something).
- In Table 2 (titled “under standard augmentation”), the improvements over baselines and vanilla-full-training are marginal.
- Why not applying with strong augmentation all the retained samples (i.e., keep + recycle)? Why applying strong augmentation only for the recycling? (This follows the first point).
- I found the notations in the equation definitions too complicated and too detailed for describing simple things that could be explained in plain text description. For example, instead of defining the scores and defining the process of the augmentations in a complex math notation, equation 1 and 2 are full of letters and indexes which I find hard to follow. Consider simplify the formulation of the method presentation.
- There is very limited improvement when scaling to ImageNet (Table 4).
- Also, seems like the accuracy improvements on object detection and semantic segmentation are marginal (Table 6 and Table 7). Indeed, the RtD saves more BP, but this narrows the claim of the method significantly.
Questions for authors
- What is the impact of using strong augmentation for all the samples, not only the recycle one?