acceptodds
Withdrawn from ICLR 2027

Recycle Then Drop: Selective Recycling for Efficient Data Pruning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training larger models on more data has improved performance but also increased training cost. Dynamic data pruning reduces this cost by scoring each sample with the current model and excluding those the model already handles easily. These pruning scores are typically computed from augmented inputs. However, a score under one augmentation does not establish that the sample has little training value under another. Direct exclusion can therefore leave useful training signal unused. To address this, we propose Recycle Then Drop (RtD), which selects a fraction of the exclusion candidates and trains them with stronger augmentation. This extends the binary pruning decision to keeping, recycling, or dropping samples. Because these decisions still require scoring every sample, we further introduce an amortized variant, A-RtD, which periodically updates and reuses them to reduce repeated scoring and input processing. Experiments across five backbones, multiple datasets, and various augmentation settings demonstrate improvements in accuracy and training efficiency. On CIFAR, RtD reduces the number of samples used for backpropagation by over 80%, while A-RtD reduces training time by up to 54%, both achieving higher accuracy than full data training. Both methods also outperform full data training on object detection and semantic segmentation while requiring less backpropagation. Additional evaluations demonstrate compatibility with existing pruning methods. These results establish selective recycling as a practical approach to improving the accuracy and efficiency of data pruning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Data pruning, Efficient training, Dynamic sample selection, Data augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fSEtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • In the paper, the authors proposed Recycle Then Drop (RtD) for dynamic data pruning, in which, within each mini-batch, instead of dropping the pruned samples, they keep a portion of them and apply them with stronger augmentation (recycle set).

  • Their default pruning score is the ground-truth-class softmax probability.

  • They show compatability to several pruning methods.

  • Also, they proposed Amortized RtD - a version that enables to reuse the scores computed for samples for later use to reduce the training time.

  • Experiments conducted on image classification tasks (CIFAR-10/100 and ImageNet-1k), object detection and semantic segmentation on PASCAL-VOC.

  • 
The RtD is compared to multiple relevant baselines and across several benchmarks.

Critical strengths
  • Broad comparison to baselines (both static and dynamic pruning) and different architectures. Adding the non-classification tasks (object detection and semantic segmentation) is useful.
  • The method was tested with several pruning approaches.
Critical weaknesses
  • It seems like the main results in the paper at Table 1 are based on unfair comparison between the RtD and the full-data baseline. While the RtD uses strong augmentation for the recycled samples (“Recycled samples additionally receive HorizontalFlip and RandAugment”), the baseline methods used only basic augmentation. Therefore, the claim that the RtD achieves higher accuracy than the full-data training is problematic (unless I miss something).
  • In Table 2 (titled “under standard augmentation”), the improvements over baselines and vanilla-full-training are marginal.
  • Why not applying with strong augmentation all the retained samples (i.e., keep + recycle)? Why applying strong augmentation only for the recycling? (This follows the first point).
  • I found the notations in the equation definitions too complicated and too detailed for describing simple things that could be explained in plain text description. For example, instead of defining the scores and defining the process of the augmentations in a complex math notation, equation 1 and 2 are full of letters and indexes which I find hard to follow. Consider simplify the formulation of the method presentation.
  • There is very limited improvement when scaling to ImageNet (Table 4).
  • Also, seems like the accuracy improvements on object detection and semantic segmentation are marginal (Table 6 and Table 7). Indeed, the RtD saves more BP, but this narrows the claim of the method significantly.
Questions for authors
  • What is the impact of using strong augmentation for all the samples, not only the recycle one?
Reviewer_cTQQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Recycle Then Drop (RtD), an extension to dynamic data pruning that preserves the original keep set, uniformly samples a fraction of exclusion candidates, and trains those candidates with stronger augmentation. Its amortized variant, A-RtD, periodically refreshes pruning decisions and reuses them between refreshes to reduce scoring and input-processing costs. The evaluation covers CNN and Transformer backbones on CIFAR, ImageNet-1K, object detection, semantic segmentation, and additional optimizer, fine-grained, and long-tailed settings. The paper reports substantial reductions in backpropagated sample usage and measured training time, supported by candidate-selection, augmentation, gradient, and caching diagnostics.

Critical strengths
  1. A simple pruning extension with demonstrated practical efficiency. RtD provides a straightforward mechanism for recovering training signal from exclusion candidates while preserving existing keep decisions. The paper explicitly distinguishes Effective Epochs (EE) from wall-clock savings and provides a component-level runtime breakdown. On ImageNet-1K, A-RtD reduces reported training time from 30.98 to 20.80 hours, approximately 32.9%, while maintaining accuracy above the full-data baseline. These measurements establish practical value beyond reducing backpropagated sample counts.

  2. Broad empirical evaluation with useful controls. The experiments cover multiple architectures, optimizers, and visual recognition tasks, including long-tailed classification. The identical-augmentation, candidate-pool, augmentation-placement, and gradient norm-matching controls provide evidence that selective recycling offers benefits beyond simply introducing a stronger augmentation policy. Although their coverage is limited, these controls meaningfully strengthen the empirical case for the method.

Critical weaknesses
  1. The comparative evaluation does not sufficiently establish the advantage of selective recycling under matched augmentation and computational budgets. In the main experiments, recycled samples receive additional augmentation, and adding recycling also increases sample usage relative to the underlying pruning method. For example, Table 5 improves DBPD from 74.34% to 76.19% with ordinary recycling and to 76.63% with stronger augmentation, while EE increases from 24.7 to 35.4 and 36.9. These results demonstrate useful improvements, but do not establish whether recycling is preferable to less aggressive pruning at a comparable budget. Tables 15 and 18 partially address this concern, but the former covers one dataset/backbone and the latter studies augmentation placement at r = 1, rather than the default r = 0.1. A cached binary-pruning control would also help separate the benefits of decision reuse and recycling. Furthermore, the comparisons omit the recent dynamic pruning baseline B-PAS [1] and the closely related joint selection-and-augmentation method of Yang et al. [2], although the latter is cited.

  2. Statistical reporting and evidence for transferable settings remain incomplete. The main tables do not consistently specify seed counts or report uncertainty estimates. This matters particularly for the ImageNet accuracy improvements of 0.25 and 0.22 percentage points over Vanilla. Table 24 provides three-seed CIFAR results, but omits Vanilla and competitive methods such as IES and DivBS. Although recycling-ratio and confidence-threshold ablations are provided, the refresh period KK is not systematically varied, despite Table 13 showing approximately 17%–25% disagreement between cached and fresh decisions early in training. The validation procedure used to select the defaults is also unclear. Finally, the encouraging aggregate long-tailed results do not reveal whether confidence-based routing changes class exposure or minority-class performance. These gaps limit confidence in the reproducibility and transferability of the reported advantages.

References:

[1] Alam, Islam, and Islam. Batch Pruning by Activation Stability. ICLR 2026. https://proceedings.iclr.cc/paper_files/paper/2026/hash/380afe1a245a3b2134010620eae88865-Abstract-Conference.html

[2] Yang, Ye, Shen, and Zhou. When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration. ICML 2025, PMLR 267:71508–71520. https://proceedings.mlr.press/v267/yang25as.html

Questions for authors
  1. Matched controls: Can you provide a focused comparison under a common augmentation recipe between RtD, binary pruning with its pruning controls adjusted to match RtD’s EE, and full-data training with a learning-rate schedule adapted to the available budget? Please consider matched EE and matched wall-clock budgets separately. How does A-RtD compare with cached binary pruning without recycling, such as K = 5, r = 0? These comparisons would clarify the respective contributions of recycling, augmentation allocation, and decision reuse.

  2. Missing comparisons and efficiency accounting: How do RtD and A-RtD compare with B-PAS [1] and Yang et al. [2] under matched training recipes, including scoring and preprocessing overhead? For interpretable data-savings comparisons, please also report a sample-weighted analogue of DSI: 100 × (1 − EE / T)%, where T is the scheduled number of full-data training epochs, counting recycled samples and full-data revision in EE. Table 2 implies 82.5% and 81.0% backpropagated-sample savings for RtD and A-RtD on CIFAR-100/ResNet-50; these should remain clearly distinguished from forward-pass and runtime savings.

  3. Statistical reliability and selection protocol: What are the seed counts and aggregation conventions for the main tables? Can you provide repeated-run results for the principal ImageNet and CIFAR comparisons, including Vanilla and the strongest competing methods? Please also specify the validation data and procedure used to select τ, r, K, augmentation settings, and detection/segmentation checkpoints.

  4. Hyperparameter transfer: How do accuracy and runtime vary with the refresh period KK, particularly given the early disagreement in Table 13? How sensitive are the recycling ratio and confidence threshold to backbone and batch size, considering that sampling ⌊r|E_{t,b}|⌋ candidates makes the realized recycling fraction depend on the candidate-pool size?

  5. Class coverage: Can you report class-wise keep/recycle/drop frequencies and per-class or head/medium/tail accuracy for the long-tailed experiments? This would clarify whether the aggregate improvements are accompanied by preserved or improved minority-class exposure and performance.

  6. Implementation clarification: Please specify Table 15’s complete augmentation pipeline and its relationship to Table 2, the padding used for RandomCrop(224) in Table 11, BatchNorm behavior during scoring and selected-sample updates, and how InfoBatch’s rescaling weights are handled when recycling is added. These details affect both reproducibility and interpretation of the controls.

Reviewer_CJeaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that dynamic data-pruning scores are computed on one augmented view, so a drop decision does not imply that the sample is redundant under another augmentation. RtD therefore extends the binary keep/drop decision to keep/recycle/drop: it keeps the original pruning rule's keep set, uniformly recycles a fraction r of the drop candidates, trains them with stronger augmentation in the same update, and skips back-propagation for the rest. A-RtD amortizes pruning decisions by refreshing them every K epochs and reusing cached decisions in between, avoiding repeated scoring and input processing. Experiments cover five backbones on CIFAR-10/100, ImageNet-1k, VOC detection/segmentation, fine-grained and long-tailed settings, four optimizers, and compatibility with InfoBatch/IES/DBPD, reporting >80% fewer back-propagated samples with accuracy above full-data training and up to 54% wall-clock reduction.

Critical strengths

The observation "augmentation-level redundancy ≠ sample-level redundancy" is crisp and leads to a simple, practical three-way routing rule that does not modify the underlying selector. (2) The engineering is solid: complete algorithm, overhead breakdown, cached-vs-fresh decision disagreement analysis (~2% in the final third), and wall-clock numbers rather than only effective epochs. (3) The control experiments arethoughtful for this literature — identical augmentation , gradient-norm matching, augmentation placement etc.

Critical weaknesses

(1) The headline "better than full-data training" is measured against a weak baseline: CIFAR-10 EfficientNet-B0 vanilla is 94.37 and CIFAR-100 is 76.22, while the authors' own SGD run (Table 23) reaches 95.50/79.77 on the same architecture. The comparison therefore confounds recycling with an under-tuned recipe. (2) No compute-matched control: RtD trains on ~12% of the samples, so the natural baseline is full data trained for the same wall-clock/EE, which is not reported. (3) Main tables (1, 2, 4) are single-seed, and several conclusions rest on 0.2–0.4-point differences. (4) The method changes selection, augmentation and loss normalization simultaneously; the controlled evidence for the augmentation confound lives in one appendix setting without variance, while the main tables still give recycled samples RandAugment+RE that vanilla never sees. (5) Effective epochs overstate the efficiency gain because scoring dominates wall-clock (EE −80% vs time −25%); the A-RtD gain largely comes from skipping scoring, not from recycling.

Questions for authors

the headline accuracy claim is anchored on an under-tuned baseline: on CIFAR-100/ResNet-50/200 epochs the full-data accuracy is 80.6 in OrderDP's paper and 81.1 in ES's, versus 76.39 in this paper's default AdamW recipe (79.77 with SGD and 82.22 with LARS in the authors' own Table 23). The main tables are single-seed, there is no compute-matched full-data baseline, Tables 3 and 14 report times on different GPUs, and Eq. (6) mixes the keep and recycled means with |H|/|U| and |R|/|U| weights without any bias correction, so recycling is a biased update by the standard set by InfoBatch and OrderDP. Re-running Table 2/24 under the OneCycle+SGD recipe with seeds, adding an OrderDP compatibility experiment, and separating the gain of recycling from the gain of skipping scoring are needed before the claim stands.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.