- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SPADE adapts the sampling distribution during training by propagating learning signals from the current mini-batch to nearby examples in a cached representation space. Its signal is the norm of the prediction error at the logits, closely related to EL2N. Neighbor priorities are updated using similarity, sender and receiver corrections, and a recovery mechanism. The paper evaluates the method on image classification, semantic segmentation, and instruction tuning.
Critical strengths
None
Critical weaknesses
-
The propagation rule relies on several heuristic choices, and the evidence does not sufficiently explain when these choices work. The basic assumption is that a sample with a large prediction error identifies a region that deserves more training. But a large error does not necessarily mean that additional training will be useful: it can also arise from incorrect labels, ambiguity, or examples the current model cannot fit effectively. Propagating that signal makes a further assumption that nearby examples share its learning value.
-
The sender and receiver corrections regulate the amount of propagation, but they do not directly establish whether the source signal should be propagated. The full update combines several choices: top-k neighborhoods, log-relative error, a neighborhood-support correction, an incoming-degree correction, an exponential update, and recovery toward uniform weights. The individual motivations are understandable, but the paper does not sufficiently establish why this combination is preferable to simpler alternatives.
-
I acknowledge the component ablations and the experiments with label noise and visual corruption. These provide useful evidence and should not be dismissed. However, they do not directly show which samples receive additional exposure, whether noisy examples are repeatedly prioritized, or whether cached neighbors reliably share the anchor’s learning needs.
-
Neighborhood size also has a substantial effect in some settings. At the 30% budget on Tiny-ImageNet, accuracy decreases from 45.9% with k=128 to 41.3% with k=1024. The authors acknowledge that there is no general rule for choosing k. This makes the transferability of the configuration an important unresolved question.
-
The practical efficiency claim needs stronger comparisons and larger data pools. Dynamic random sampling is the relevant simple baseline here. At the 30% budget on Tiny-ImageNet, changing static random selection to dynamic random selection already increases accuracy from 29.8% to 41.8%, while SPADE reaches 44.9%. SPADE therefore has an additional benefit, but much of the gap over static selection comes from revisiting the choice of data rather than from the proposed propagation mechanism.
-
The remaining question is whether that additional benefit survives a matched wall-clock comparison. SPADE maintains a dataset-wide feature cache and performs exhaustive neighborhood retrieval with per-step cost O(BNd). Dynamic random sampling avoids this machinery and could spend the saved time on more training. The efficiency plots compare SPADE with InfoBatch and Data Agent, rather than directly resolving this comparison with dynamic random sampling or ordinary full-data training. The headline time reduction should be interpreted within that limited comparison.
-
The evaluated candidate pools also do not adequately test this scaling bottleneck. Tiny-ImageNet contains only 100k training images, and the instruction-tuning experiment uses the small Alpaca dataset. Using a 7B model does not establish that dataset-wide retrieval scales to a large corpus. The proposed ANN acceleration is an estimate rather than an implemented and measured result. I would need stronger evidence before accepting the method as a practical approach to large-scale training efficiency.
Questions for authors
-
How does SPADE compare with dynamic random sampling under the same total wall-clock budget, allowing random sampling to use its lower overhead for additional training? Please also specify the resampling frequency and learning-rate schedule used for Random*.
-
Can the authors isolate the benefit of neighborhood propagation by comparing it with direct online sampling from cached per-example error scores, using the same weighting, recovery, and training budget? An exposure analysis separating clean, noisy, easy, and difficult examples would also help explain the mechanism.
-
What are the measured time and memory costs of cache initialization, retrieval, cache refresh, and sampling as the candidate pool grows? How are never-visited samples initialized, and how sensitive are results to stale features and transferring k between datasets?
Thank you for the specific questions. We now quantify neighborhood sensitivity, distinguish identity-level retrieval mismatch from class/residual consistency, and isolate recovery while reporting actual million-example training.
W1: Difficulty and neighborhood transfer
Prior work supports prediction error, forgetting, and uncertainty as useful selection signals (Paul et al.; Toneva et al.; He et al.), and D²-Pruning already aggregates difficulty over neighboring examples. SPADE extends this idea from static pruning to online exposure, prioritizing currently difficult regions while calibrating how much signal each transfer carries. The sender coefficient ψi, a bell-shaped function of log effective neighborhood support, dampens anchors with atypically low support (often isolated high-error examples) or atypically high support (which would spread one signal across loosely related candidates). The receiver coefficient νj normalizes by incoming degree, so frequently retrieved candidates are not amplified by connectivity alone.
W2 · Q2: Matched direct cached-error control
We thank the reviewer for this suggestion. To isolate the benefit of neighborhood propagation, we compare SPADE against a direct cached-error controller that updates each observed example's own sampling priority from its normalized residual, without propagation. Both use the same weighting, recovery, optimizer, training schedule, and presentation budget.
Under these matched conditions, SPADE improves accuracy by 1.4 and 2.0 percentage points, respectively. These results support the benefit of transferring online feedback to neighboring examples beyond updating only the observed examples' priorities.
W4 · Q3: Neighborhood-size sensitivity
On CIFAR-100, k=24 performs better at 10–20% budgets, while k=64 performs better at 30–70%. The accuracy difference never exceeds 0.7 points. Thus, the preferred k varies with budget, although both tested choices remain competitive. On Tiny-ImageNet, overly large neighborhoods (k≥512) reduce accuracy, which we will state explicitly in the revision. We do not claim a universally optimal or transferable k.
W5 · W6 · Q1: Dynamic random and wall-clock efficiency
Random* follows the dynamic-random baseline of InfoBatch (Qin, Ziheng, et al). In the main Tiny-ImageNet table, SPADE's reported gains are +2.0/+7.3/+3.1/+1.5/+1.7 points at 10/20/30/50/70% selection.
We use SPADE's training-completion time as the shared wall-clock budget and allow dynamic random sampling to continue training until the same cutoff, spending its lower overhead on additional training.
Even when dynamic random can use the saved time for additional training, SPADE achieves 1.37 percentage points higher accuracy, respectively.
W7 · Q3: Initialization, scalability, and stale features
SPADE initializes its representation cache during the first full-data training epoch, reusing features from standard training forward passes without requiring additional full-dataset inference. Cached embeddings are subsequently refreshed whenever samples are revisited, allowing the representation space to evolve alongside model optimization.
To further test this, our ImageNet-1K profiling experiment, involving 1.28 million training samples, evaluates the computational overhead of exact k-nearest-neighbor retrieval, cache maintenance, learning-signal propagation, recovery, and sampling. Together, these operations account for 10.30% of recorded training time. The FP32 feature bank requires approximately 9.77 GiB for 2,048-dimensional embeddings, excluding temporary buffers and model memory. These results demonstrate the practical feasibility of SPADE's exact retrieval mechanism at million-example scale.