acceptodds
Under review as a conference paper at ICLR 2027

From Selection to Exposure: Online Data Selection via Learning-Signal Propagation

14 views

Abstract

Data selection aims to reduce the cost of training by allocating computation toward informative examples, while maintaining comparable performance as example utility changes throughout learning. Existing methods estimate sample importance using early-training statistics, losses, or learned selection policies and then prune or prioritize a subset of the training data. While effective, these approaches can over-concentrate training on currently prioritized samples, reducing coverage of other regions of the training distribution. We propose an online data-selection framework that turns each mini-batch into feedback for selecting subsequent training examples. At each optimization step, feedback from the current batch continuously reshapes the sampling distribution over nearby examples in the evolving representation space, increasing exposure to currently informative regions without deterministically excluding lower-priority samples. This allows the current batch to directly influence future training exposure while preserving broader data coverage and reusing representations naturally produced during training. Across image classification, semantic segmentation and instruction tuning, SPADE achieves strong predictive performance with high training efficiency. On Tiny-ImageNet, SPADE reaches comparable accuracy with up to 67.2% fewer training sample presentations and 64.1% less wall-clock time. Across the three image-classification benchmarks, it achieves the best Top-1 accuracy in 12 of the 15 settings evaluated. On ADE20K, SPADE reaches 42.0% mIoU using 70% of the training data, improving over the compared vanilla baseline. For LLaMA-7B instruction tuning using only a small fraction of Alpaca, SPADE also improves over the compared baselines, achieving the highest average score of 26.5 across BBH, DROP, MMLU and HumanEval.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Dynamic Data Selection, Data Selection, Training acceleration, Dataset Pruning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dUjLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SPADE adapts the sampling distribution during training by propagating learning signals from the current mini-batch to nearby examples in a cached representation space. Its signal is the norm of the prediction error at the logits, closely related to EL2N. Neighbor priorities are updated using similarity, sender and receiver corrections, and a recovery mechanism. The paper evaluates the method on image classification, semantic segmentation, and instruction tuning.

Critical strengths

None

Critical weaknesses
  • The propagation rule relies on several heuristic choices, and the evidence does not sufficiently explain when these choices work. The basic assumption is that a sample with a large prediction error identifies a region that deserves more training. But a large error does not necessarily mean that additional training will be useful: it can also arise from incorrect labels, ambiguity, or examples the current model cannot fit effectively. Propagating that signal makes a further assumption that nearby examples share its learning value.

  • The sender and receiver corrections regulate the amount of propagation, but they do not directly establish whether the source signal should be propagated. The full update combines several choices: top-k neighborhoods, log-relative error, a neighborhood-support correction, an incoming-degree correction, an exponential update, and recovery toward uniform weights. The individual motivations are understandable, but the paper does not sufficiently establish why this combination is preferable to simpler alternatives.

  • I acknowledge the component ablations and the experiments with label noise and visual corruption. These provide useful evidence and should not be dismissed. However, they do not directly show which samples receive additional exposure, whether noisy examples are repeatedly prioritized, or whether cached neighbors reliably share the anchor’s learning needs.

  • Neighborhood size also has a substantial effect in some settings. At the 30% budget on Tiny-ImageNet, accuracy decreases from 45.9% with k=128 to 41.3% with k=1024. The authors acknowledge that there is no general rule for choosing k. This makes the transferability of the configuration an important unresolved question.

  • The practical efficiency claim needs stronger comparisons and larger data pools. Dynamic random sampling is the relevant simple baseline here. At the 30% budget on Tiny-ImageNet, changing static random selection to dynamic random selection already increases accuracy from 29.8% to 41.8%, while SPADE reaches 44.9%. SPADE therefore has an additional benefit, but much of the gap over static selection comes from revisiting the choice of data rather than from the proposed propagation mechanism.

  • The remaining question is whether that additional benefit survives a matched wall-clock comparison. SPADE maintains a dataset-wide feature cache and performs exhaustive neighborhood retrieval with per-step cost O(BNd). Dynamic random sampling avoids this machinery and could spend the saved time on more training. The efficiency plots compare SPADE with InfoBatch and Data Agent, rather than directly resolving this comparison with dynamic random sampling or ordinary full-data training. The headline time reduction should be interpreted within that limited comparison.

  • The evaluated candidate pools also do not adequately test this scaling bottleneck. Tiny-ImageNet contains only 100k training images, and the instruction-tuning experiment uses the small Alpaca dataset. Using a 7B model does not establish that dataset-wide retrieval scales to a large corpus. The proposed ANN acceleration is an estimate rather than an implemented and measured result. I would need stronger evidence before accepting the method as a practical approach to large-scale training efficiency.

Questions for authors
  1. How does SPADE compare with dynamic random sampling under the same total wall-clock budget, allowing random sampling to use its lower overhead for additional training? Please also specify the resampling frequency and learning-rate schedule used for Random*.

  2. Can the authors isolate the benefit of neighborhood propagation by comparing it with direct online sampling from cached per-example error scores, using the same weighting, recovery, and training budget? An exposure analysis separating clean, noisy, easy, and difficult examples would also help explain the mechanism.

  3. What are the measured time and memory costs of cache initialization, retrieval, cache refresh, and sampling as the candidate pool grows? How are never-visited samples initialized, and how sensitive are results to stale features and transferring k between datasets?

1 comment

Thank you for the specific questions. We now quantify neighborhood sensitivity, distinguish identity-level retrieval mismatch from class/residual consistency, and isolate recovery while reporting actual million-example training.

W1: Difficulty and neighborhood transfer

Prior work supports prediction error, forgetting, and uncertainty as useful selection signals (Paul et al.; Toneva et al.; He et al.), and D²-Pruning already aggregates difficulty over neighboring examples. SPADE extends this idea from static pruning to online exposure, prioritizing currently difficult regions while calibrating how much signal each transfer carries. The sender coefficient ψi\psi_i, a bell-shaped function of log effective neighborhood support, dampens anchors with atypically low support (often isolated high-error examples) or atypically high support (which would spread one signal across loosely related candidates). The receiver coefficient νj\nu_j normalizes by incoming degree, so frequently retrieved candidates are not amplified by connectivity alone.

W2 · Q2: Matched direct cached-error control

We thank the reviewer for this suggestion. To isolate the benefit of neighborhood propagation, we compare SPADE against a direct cached-error controller that updates each observed example's own sampling priority from its normalized residual, without propagation. Both use the same weighting, recovery, optimizer, training schedule, and presentation budget.

Dataset / selection ratioDirect cached errorSPADEGain
CIFAR-100 / 30%76.177.5+1.4
Tiny-ImageNet / 20%40.942.9+2.0

Under these matched conditions, SPADE improves accuracy by 1.4 and 2.0 percentage points, respectively. These results support the benefit of transferring online feedback to neighboring examples beyond updating only the observed examples' priorities.

W4 · Q3: Neighborhood-size sensitivity

On CIFAR-100, k=24k=24 performs better at 10–20% budgets, while k=64k=64 performs better at 30–70%. The accuracy difference never exceeds 0.7 points. Thus, the preferred kk varies with budget, although both tested choices remain competitive. On Tiny-ImageNet, overly large neighborhoods (k≥512k \geq 512) reduce accuracy, which we will state explicitly in the revision. We do not claim a universally optimal or transferable kk.

W5 · W6 · Q1: Dynamic random and wall-clock efficiency

Random* follows the dynamic-random baseline of InfoBatch (Qin, Ziheng, et al). In the main Tiny-ImageNet table, SPADE's reported gains are +2.0/+7.3/+3.1/+1.5/+1.7 points at 10/20/30/50/70% selection.

We use SPADE's training-completion time as the shared wall-clock budget and allow dynamic random sampling to continue training until the same cutoff, spending its lower overhead on additional training.

DatasetShared wall-clock budgetSPADEDynamic random
CIFAR-100 / 30%320 s77.576.13

Even when dynamic random can use the saved time for additional training, SPADE achieves 1.37 percentage points higher accuracy, respectively.

W7 · Q3: Initialization, scalability, and stale features

SPADE initializes its representation cache during the first full-data training epoch, reusing features from standard training forward passes without requiring additional full-dataset inference. Cached embeddings are subsequently refreshed whenever samples are revisited, allowing the representation space to evolve alongside model optimization.

To further test this, our ImageNet-1K profiling experiment, involving 1.28 million training samples, evaluates the computational overhead of exact kk-nearest-neighbor retrieval, cache maintenance, learning-signal propagation, recovery, and sampling. Together, these operations account for 10.30% of recorded training time. The FP32 feature bank requires approximately 9.77 GiB for 2,048-dimensional embeddings, excluding temporary buffers and model memory. These results demonstrate the practical feasibility of SPADE's exact retrieval mechanism at million-example scale.

Reviewer_x8YsOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SPADE, an online data selection method that continuously adjusts training exposure at the mini batch level. SPADE computes a prediction based learning signal for observed examples, propagates it to nearby candidates in the model’s evolving representation space, and updates their future sampling probabilities while keeping all examples reachable. The method is evaluated on image classification, semantic segmentation, and instruction tuning, with additional experiments on corrupted data, component ablations, and training efficiency.

Critical strengths
  1. Adaptive exposure is a useful formulation. Maintaining a soft, step level sampling distribution allows priorities to evolve during training without permanently excluding lower priority examples. This is a practically appealing alternative to fixed coreset selection.

  2. The image classification evaluation is reasonably broad. The paper compares SPADE with numerous static and dynamic baselines across three datasets and five selection ratios, with additional corruption and runtime experiments. These results provide some evidence that adaptive exposure can be effective under reduced training budgets.

Critical weaknesses
  1. The central contribution is not adequately isolated. The proposed learning signal is algebraically equivalent to the existing EL2N score, making online neighborhood propagation the main novel component. However, the paper provides no ablation that removes propagation while retaining the same signal, sampling schedule, and reweighting. The reported gains therefore cannot be attributed specifically to the proposed mechanism.

  2. The implementation and scalability of propagation are insufficiently established. The paper does not explain how embeddings are initialized for candidates that have not yet been observed, even though neighbor retrieval requires a complete representation cache. Moreover, the current implementation performs exhaustive full bank retrieval at every step and is evaluated only on modestly sized datasets, while large scale ANN performance is supported solely by theoretical estimates. These limitations undermine both reproducibility and the claimed applicability to large scale data selection.

Questions for authors
  1. How is the representation cache initialized for training examples that have not yet been observed, particularly when the target model is trained from scratch? Does initialization require an additional full dataset forward pass, and is this cost included in the reported runtime?

  2. How does SPADE compare with an otherwise identical variant that disables neighborhood propagation by setting α=0\alpha=0 while retaining the same EL2N signal, sampling schedule, recovery, and importance weighting?

  3. Do the authors have empirical results using approximate neighbor retrieval on a substantially larger data pool? If not, how should the current scalability claims be interpreted given that the reported experiments use exhaustive full bank retrieval?

2 comments

W2 · Q1: Cache initialization

For the from-scratch vision experiments, the first epoch trains on the full training set. No additional full-dataset forward pass is performed. Cache initialization reuses the first training epoch, which is included in the stated training horizon. Its full-data computation and cache writes are included in the logged training/runtime totals and sample-presentation counts; they are not treated as free preprocessing.

W2 · Q3: Scalability and ANN retrieval

The primary scope of this work is to investigate the effectiveness of online learning-signal propagation for adaptive data selection, rather than to develop or benchmark approximate nearest-neighbor (ANN) algorithms. Appendix B.3 already provides a theoretical complexity analysis demonstrating the potential computational advantage of ANN over exhaustive retrieval, with estimated reductions of 83.3× at one million candidates under the stated assumptions. These estimates are explicitly theoretical, not measured speedups. We therefore consider additional ANN implementation and benchmarking beyond the scope of this paper, as they are not necessary to establish the proposed selection mechanism or its feasibility at the evaluated scale. We will clarify this distinction between demonstrated scalability and theoretical acceleration.

Q2 · W1: Isolating neighborhood propagation

Our formulation provides an explicit difficulty-based interpretation of the signal through GiG_i, separating error magnitude from the concentration of probability mass among incorrect classes. For one-hot targets, this quantity has the same algebraic form as the per-model EL2N norm. Unlike the original EL2N protocol, which aggregates early scores to select a fixed subset, SPADE uses current, normalized difficulty signals throughout training and propagates them to related candidates before they are revisited.

Setting α=0\alpha = 0 eliminates neighborhood propagation: priorities stay uniform, and Eq. (17) reduces to uniform dynamic sampling without replacement, which connects directly to the Random* baseline.

In Table 1 of the manuscript, SPADE outperforms Random* in all 15 settings and static EL2N in 14, with one tie, and achieves the highest accuracy among all compared methods in 12 of 15. At the 30% budget, SPADE improves over Random* by 2.6 points on CIFAR-100 (77.5 vs. 74.9) and 3.1 points on Tiny-ImageNet (44.9 vs. 41.8).

Table 5 isolates the calibration mechanisms on CIFAR-100: removing sender reliability (ψ\psi), receiver correction (ν\nu), or both lowers accuracy from 77.5% to 75.6%, 76.3% and 74.8%, respectively (−1.9, −1.2 and −2.7 points). These results support the importance of calibrating outgoing and incoming learning signals within the propagation framework.

Together, Table 1 demonstrates the effectiveness of SPADE relative to the reported EL2N and dynamic-random baselines, while Table 5 provides component-level evidence that sender and receiver calibration improve the propagation mechanism.

Q3: Approximate retrieval on a larger pool

Following the reviewer's suggestion, we have added an ImageNet-1K experiment (1.28M training images, roughly 13× larger than Tiny-ImageNet), using exact top-kk retrieval over the full feature bank.

ImageNet-1K (1.28M training images): SPADE top-1 accuracy (%) by selection ratio.

Selection ratio10%20%30%50%60%
SPADE70.573.274.975.676.5

For reference, the table below places SPADE alongside ImageNet-1K ResNet-50 results reported by prior dynamic data-selection methods. These numbers are taken from the respective papers under their own training setups, so they serve as a reference point rather than a controlled comparison.

Method30%~60%70%
UCB (Raju et al., 2021)68.59 (a)75.8 (b)72.97 (a)
ϵ\epsilon-greedy (Raju et al., 2021)69.28 (a)–74.10 (a)
InfoBatch (Qin et al., 2024)72.51 (a)76.5 (b)74.59 (a)
TED (Xiao et al., ECAI 2024)72.68 (a)–74.67 (a)
SPADE (ours)74.976.5–

(a) Reported in TED, Table 3; full-data baseline 75.28. (b) Reported in InfoBatch, Table 2 (90 epochs, about 40% of samples pruned); full-data baseline 76.4.

At the 30% budget, SPADE reaches 74.9%, above the reported InfoBatch (72.51%) and TED (72.68%) results. At about 60%, SPADE matches InfoBatch's reported 76.5%, which is on par with full-data training (76.4%) in that setup.

Thank you for the detailed response and additional experiments. The cache clarification, ImageNet-scale profiling, and matched cached-error control address parts of my implementation and ablation concerns. However, the matched propagation control is limited to two vision settings, while the ImageNet comparison is not controlled and the broader evidence, particularly for instruction tuning, remains insufficient to establish the general value of the heuristic propagation design. I therefore maintain my rating.

Reviewer_qmBeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This submission introduces SPADE, an online data-selection framework that transforms each mini-batch into feedback to continuously reshape a time-varying sampling distribution over the training set. Unlike static pruning methods that make hard keep-or-discard decisions, this approach estimates per-example learning signals by combining target difficulty with prediction concentration and propagates these signals through representation-space neighborhoods to adjust the future training exposure of related unobserved candidates. By incorporating sender reliability and receiver balance coefficients to calibrate the propagation, SPADE ensures that informative regions receive more computation while lower priority samples remain reachable, achieving superior training efficiency and predictive performance across image classification, semantic segmentation, and instruction tuning tasks.

Critical strengths
  1. Dynamic and Soft Adaptive Exposure: Unlike traditional static pruning that relies on irreversible hard keep-or-discard decisions, SPADE maintains a time-varying sampling distribution that remains responsive to the evolving model state. This soft selection mechanism ensures that lower priority samples are never permanently excluded from the training support, thereby preserving broader data coverage and allowing previously easy examples to regain exposure if they become informative again as the decision boundary shifts.

  2. Semantic Learning Signal Propagation: The framework introduces a propagation mechanism that transfers optimization feedback from observed mini-batches to related unobserved candidates via representation-space neighborhoods. By integrating target difficulty with a concentration signal and further calibrating this transfer through sender reliability and receiver balance coefficients, the method effectively identifies and prioritizes informative regions without requiring auxiliary selection models or expensive per-sample gradient computations.

Critical weaknesses
  1. The selection of the neighborhood size k lacks a principled formulation and remains sensitive to the dataset size N and the selection rate. Table 6 shows that while k=64 is optimal for CIFAR-100, performance fluctuates across different settings, suggesting that the propagation mechanism might lead to signal dilution or insufficient coverage if k is not manually tuned for each specific training budget.

  2. Reliance on Evolving Representation Quality: The method assumes that the representation space produced by the task model is sufficiently discriminative to identify semantically related candidates. However, in the early stages of training or under very low selection rates (e.g., 10%), the model may not develop reliable embeddings, causing the propagation of learning signals to irrelevant candidates and reinforcing poor sampling biases.

  3. While SPADE performs well on benchmarks like Tiny-ImageNet, the core argument regarding concentration signals and receiver balance is most relevant for large-scale datasets with significant redundancy. The current experiments do not fully explore whether the multiplicative priority update wjexp⁡(Δj)w_j \exp(\Delta_j) remains stable when the number of incoming neighborhood connections deg⁡T−(j)\deg_T^{-}(j) scales to millions of samples.

Questions for authors
  1. For weekness1, could the authors provide a sensitivity analysis on how the optimal k evolves as the training budget decreases from 70% to 10%, and answer whether a fixed k has potential risks the learning signals in low-data regimes?

  2. For weekness 2, since the cache is refreshed only when a sample is revisited, how does the "temporal mismatch" between the cached features and the current model state affect the accuracy of neighborhood retrieval during the highly non-linear early phase of optimization?

  3. For weekness 3, could the authors clarify if the priority recovery mechanism in Equation (16) is sufficient to prevent the sampling distribution from becoming overly peaked, or does it eventually require a more aggressive reset to maintain the reachability of easy samples?

1 comment

W1 · Q1: Sensitivity to neighborhood size and training budget

We agree that the preferred neighborhood size can vary with the training budget. Our CIFAR-100 sensitivity analysis reports:

Selection ratiok=24k=24k=64k=64
10%72.671.9
20%76.576.4
30%77.277.5
50%77.978.2
70%79.079.2

Among these two choices, the preferred kk shifts from 64 to 24 as the budget decreases. Nevertheless, the accuracy gap remains within 0.7 percentage points across all budgets. Thus, a fixed k=64k=64 can modestly reduce accuracy at low budgets, but competitive performance does not require selecting a different kk for every budget. Table 6 further shows that non-optimal choices within the tested range k=64k=64–20482048 remain competitive with most baselines. These results support practical robustness within the evaluated settings, while kk remains a configuration-dependent hyperparameter.

Q2: Stale representations

SPADE addresses potentially unreliable early-stage representations and stale cached embeddings through two mechanisms. First, sender reliability (ψi\psi_i) and receiver correction (νj\nu_j) calibrate propagated learning signals rather than relying solely on embedding similarity. Second, cached representations are refreshed when samples are revisited, allowing neighborhood information to evolve alongside the model.

Our ablation study (Table 5) demonstrates the effectiveness of these corrections. Removing sender correction reduces accuracy from 77.5% to 75.6%, removing receiver correction reduces it to 76.3%, and removing both reduces it to 74.8%.

W3 · Q3: Priority recovery

Equation (16) gradually restores sampling priorities toward uniformity, preventing historical learning signals from permanently dominating future exposure. Our recovery ablation demonstrates that this mechanism substantially reduces sampling concentration while maintaining predictive performance.

Recovery configurationMinimum ESS/NLow-G sample returnAccuracy
No recovery (ρ=0\rho=0)5.75%77.68%69.84%
Gradual recovery (ρ=0.05\rho=0.05)51.92%97.04%77.50%

These results show that gradual recovery effectively balances adaptive prioritization and sample reachability, without requiring an additional hard reset in the evaluated configuration.

Priority Recovery. Equation (16) gradually restores priority weights toward uniformity, preventing historical learning signals from permanently dominating sampling. Our CIFAR-100 ablation (30% budget) shows that recovery (ρ=0.05\rho=0.05) improves accuracy from 69.84% to 77.50% (+7.66 points), increases minimum normalized effective sample size from 5.75% to 51.92%, and raises the ten-epoch return rate of low-signal samples from 77.68% to 97.04%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.